2024-09-13T13:38:10.817 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T13:38:10.840 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:38:10.937 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513 branch: wip-aclamk-cbt-combined description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} email: akupczyk@redhat.com first_in_suite: false job_id: '7903513' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 30 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - \(POOL_APP_NOT_ENABLED\) sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiZtazMfvLQKL+wFlJOffMbO0/uPP8jSwRCNXFZFASvnEXYDlLFdK3SWEJ7kBLupPB+ZRnzGuqZ9MyvqDrr8aw= smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC2wGt7rrPVKkuCK5hKUkN3SFRpUUtXtyP3pEsFzy17etv+MT0fpbkRoNkTJK2io/siRUlJXtogbxBu8hCYdvo= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLi6bw06U9TfThmHoPm/FJp9a2Q/r1TzSwY4TemdRDkkKRhWZmmpBVhh3HGeiaY1XYkod/6nniKlcWHAZOnuEk= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOejw+wAj2h+EGiSnWmxEB9kh+1PjHpuZR4oSPj+Xdtuc1VDjFPGI4k54OQOr6FDl7EDyaJHjfzJxPjXxKTchpE= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - ceph config set mgr mgr/balancer/log_level debug - ceph balancer status - ceph osd set-require-min-compat-client reef - ceph balancer mode upmap-read - ceph balancer on - ceph balancer status - ceph balancer status detail - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 map_discontinuity_sleep_time: 200 min_in: 2 timeout: 1800 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 rollback: 10 set_chunk: 30 snap_create: 10 snap_remove: 10 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T13:38:10.938 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T13:38:10.939 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T13:38:10.939 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T13:38:10.940 INFO:teuthology.task.internal:Checking packages... 2024-09-13T13:38:10.962 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T13:38:10.962 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T13:38:10.963 INFO:teuthology.packaging:ref: None 2024-09-13T13:38:10.963 INFO:teuthology.packaging:tag: None 2024-09-13T13:38:10.963 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T13:38:10.963 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:38:10.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T13:38:11.110 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T13:38:11.111 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T13:38:11.118 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T13:38:11.118 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T13:38:11.129 INFO:teuthology.task.internal:Saving configuration 2024-09-13T13:38:11.146 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T13:38:11.153 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T13:38:11.187 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:32:03.120866', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiZtazMfvLQKL+wFlJOffMbO0/uPP8jSwRCNXFZFASvnEXYDlLFdK3SWEJ7kBLupPB+ZRnzGuqZ9MyvqDrr8aw='} 2024-09-13T13:38:11.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:32:03.119116', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC2wGt7rrPVKkuCK5hKUkN3SFRpUUtXtyP3pEsFzy17etv+MT0fpbkRoNkTJK2io/siRUlJXtogbxBu8hCYdvo='} 2024-09-13T13:38:11.251 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:32:03.117096', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLi6bw06U9TfThmHoPm/FJp9a2Q/r1TzSwY4TemdRDkkKRhWZmmpBVhh3HGeiaY1XYkod/6nniKlcWHAZOnuEk='} 2024-09-13T13:38:11.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:32:03.122594', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOejw+wAj2h+EGiSnWmxEB9kh+1PjHpuZR4oSPj+Xdtuc1VDjFPGI4k54OQOr6FDl7EDyaJHjfzJxPjXxKTchpE='} 2024-09-13T13:38:11.284 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T13:38:11.292 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-13T13:38:11.292 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-13T13:38:11.292 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-13T13:38:11.292 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-13T13:38:11.292 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T13:38:11.448 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7a5b6f7be0>, signals=[15]) 2024-09-13T13:38:11.449 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T13:38:11.455 INFO:teuthology.task.internal:Opening connections... 2024-09-13T13:38:11.455 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:38:11.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:38:11.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:38:11.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:38:11.623 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:38:11.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:38:11.710 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:38:11.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:38:11.792 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T13:38:11.799 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-09-13T13:38:11.818 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-09-13T13:38:11.819 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-09-13T13:38:11.878 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:38:11.879 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:38:11.880 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-09-13T13:38:11.911 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-09-13T13:38:11.930 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-09-13T13:38:11.930 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-09-13T13:38:11.988 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:38:11.989 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:38:11.990 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:38:11.990 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-09-13T13:38:12.019 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-09-13T13:38:12.038 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-09-13T13:38:12.038 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-09-13T13:38:12.096 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-09-13T13:38:12.096 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-09-13T13:38:12.096 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:38:12.097 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:38:12.098 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-09-13T13:38:12.119 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-09-13T13:38:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-09-13T13:38:12.138 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-09-13T13:38:12.196 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:38:12.197 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:38:12.198 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:38:12.198 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:38:12.198 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:38:12.198 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-09-13T13:38:12.227 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T13:38:12.237 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T13:38:12.243 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T13:38:12.243 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:38:12.246 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:38:12.248 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:38:12.250 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:38:12.267 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T13:38:12.274 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T13:38:12.274 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:38:12.305 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:38:12.307 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:38:12.310 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:38:12.411 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T13:38:12.546 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T13:38:12.561 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T13:38:12.561 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T13:38:12.561 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:38:12.562 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:38:12.562 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:38:12.562 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:38:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:38:12.580 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T13:38:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:38:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:38:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:38:12.583 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-13T13:38:12.583 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-09-13T13:38:12.583 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-13T13:38:12.638 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:38:12.638 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-503.el9.x86_64 2024-09-13T13:38:12.639 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-09-13T13:38:12.640 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:38:12.640 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.14.0-503.el9.x86_64 2024-09-13T13:38:12.640 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-09-13T13:38:12.641 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:38:12.641 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-503.el9.x86_64 2024-09-13T13:38:12.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-09-13T13:38:12.643 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:38:12.643 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-503.el9.x86_64 2024-09-13T13:38:12.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-09-13T13:38:14.935 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-13T13:38:15.144 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-13T13:38:15.213 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-13T13:38:17.165 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-09-13T13:38:19.091 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-13T13:38:23.533 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T13:38:24.305 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-13T13:38:25.101 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-13T13:38:26.747 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-13T13:38:28.506 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-13T13:38:30.123 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-13T13:38:31.544 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:38:32.427 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T13:38:34.517 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:38:34.518 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:38:34.606 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:38:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 79 M 2024-09-13T13:38:34.614 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 126 M 2024-09-13T13:38:34.615 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:38:34.821 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 261 kB/s | 23 kB 00:00 2024-09-13T13:38:35.051 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:19 2024-09-13T13:38:35.300 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:19 2024-09-13T13:38:37.049 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T13:38:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-09-13T13:38:39.056 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:38:39.056 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:38:39.141 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:38:39.147 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:38:39.147 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-09-13T13:38:39.147 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:38:39.147 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:38:39.147 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-09-13T13:38:39.148 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:38:39.149 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 79 M 2024-09-13T13:38:39.149 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 126 M 2024-09-13T13:38:39.149 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:38:39.409 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 282 kB/s | 23 kB 00:00 2024-09-13T13:38:39.776 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-13T13:38:39.817 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.5 MB 00:00 2024-09-13T13:38:40.111 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.5 MB 00:01 2024-09-13T13:38:40.185 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 40 MB/s | 35 MB 00:00 2024-09-13T13:38:40.527 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-13T13:38:40.527 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:38:40.527 INFO:teuthology.orchestra.run.smithi159.stdout:Total 57 MB/s | 79 MB 00:01 2024-09-13T13:38:41.213 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:38:41.327 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 19 kB 00:00 2024-09-13T13:38:41.470 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 136 kB/s | 19 kB 00:00 2024-09-13T13:38:41.573 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:38:41.573 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:38:42.212 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.9 MB/s | 29 MB 00:07 2024-09-13T13:38:42.531 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:38:42.531 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:38:42.774 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-13T13:38:42.889 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-13T13:38:43.854 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-13T13:38:43.854 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:38:43.854 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.5 MB/s | 79 MB 00:09 2024-09-13T13:38:44.565 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:38:44.992 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:38:44.992 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:38:45.004 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:38:45.710 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:38:45.811 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:46.009 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:38:46.010 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:38:48.180 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-13T13:38:48.348 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-09-13T13:38:48.603 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:38:48.604 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:38:49.266 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:38:49.357 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:50.129 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:38:50.129 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:38:50.218 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:38:50.230 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:38:50.230 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T13:38:50.230 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:38:50.230 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:38:50.230 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 79 M 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 126 M 2024-09-13T13:38:50.231 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:38:50.351 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:38:50.352 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:38:50.432 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 299 kB/s | 23 kB 00:00 2024-09-13T13:38:50.445 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:38:50.452 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:38:50.453 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:38:50.453 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:38:50.453 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-13T13:38:50.453 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:38:50.453 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 79 M 2024-09-13T13:38:50.454 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 126 M 2024-09-13T13:38:50.454 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:38:50.730 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 228 kB/s | 23 kB 00:00 2024-09-13T13:38:50.982 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-13T13:38:51.113 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-13T13:38:51.422 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 41 MB/s | 29 MB 00:00 2024-09-13T13:38:51.441 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 32 MB/s | 35 MB 00:01 2024-09-13T13:38:51.814 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-13T13:38:51.815 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:38:51.815 INFO:teuthology.orchestra.run.smithi100.stdout:Total 58 MB/s | 79 MB 00:01 2024-09-13T13:38:52.217 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 16 MB/s | 29 MB 00:01 2024-09-13T13:38:52.217 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:38:52.217 INFO:teuthology.orchestra.run.smithi122.stdout:Total 40 MB/s | 79 MB 00:01 2024-09-13T13:38:52.274 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:52.408 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:38:52.496 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:38:52.884 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:38:52.884 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:38:52.892 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:38:53.234 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:38:53.234 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:38:53.566 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:38:53.607 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:38:53.830 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:38:53.830 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:38:54.208 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:38:54.208 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:38:56.278 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:38:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:38:56.964 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:38:57.048 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:57.327 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:38:57.403 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:57.633 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:38:57.676 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:38:58.452 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:38:59.840 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:38:59.977 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:00.182 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:39:00.305 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:02.914 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:04.824 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:04.935 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:05.137 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:05.180 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:10.021 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:36.675 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:36.714 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:37.090 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:37.090 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:39:37.090 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:39:37.090 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:37.934 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:39:37.935 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:39:38.306 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:38.400 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:38.400 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:38.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:39:38.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:39:38.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:39:38.401 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-13T13:39:38.401 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:39:38.401 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:38.459 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:38.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:38.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.06 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 82.el9 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Group : Unspecified 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 8274670 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-09-13T13:39:38.499 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : builder@centos.org 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:39:38.500 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-09-13T13:39:38.501 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:39:38.501 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:39:38.501 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:39:38.502 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:39:38.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:39:39.181 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-09-13T13:39:40.517 INFO:teuthology.orchestra.run.smithi159.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:39:40.555 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-09-13T13:39:40.558 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-09-13T13:39:40.573 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.EVpbqpmeJN 2024-09-13T13:39:40.574 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EVpbqpmeJN 2024-09-13T13:39:40.648 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.EVpbqpmeJN 2024-09-13T13:39:40.787 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.EVpbqpmeJN is 6KB 2024-09-13T13:39:40.836 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.EVpbqpmeJN 2024-09-13T13:39:40.852 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:39:40.915 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:39:40.915 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:39:40.915 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:39:40.915 INFO:teuthology.orchestra.run.smithi159.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:39:40.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:39:41.058 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-09-13T13:39:42.930 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:42.971 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:43.317 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:43.318 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:39:43.318 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:39:43.318 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:44.167 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:39:44.168 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:39:44.538 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:44.620 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.620 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:44.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:39:44.621 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:39:44.621 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:39:44.621 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T13:39:44.621 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:39:44.621 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:44.682 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:44.682 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:44.682 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 82.el9 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-09-13T13:39:44.724 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8274670 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-09-13T13:39:44.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:39:44.726 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:39:44.727 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:39:44.727 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:39:44.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:39:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-09-13T13:39:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:39:46.671 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-09-13T13:39:46.673 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-13T13:39:46.689 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.63AvXn4lfr 2024-09-13T13:39:46.689 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.63AvXn4lfr 2024-09-13T13:39:46.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.63AvXn4lfr 2024-09-13T13:39:46.895 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.63AvXn4lfr is 6KB 2024-09-13T13:39:46.943 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.63AvXn4lfr 2024-09-13T13:39:46.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:39:47.029 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:39:47.030 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:39:47.030 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:39:47.030 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:39:47.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:39:47.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-09-13T13:39:48.142 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.190 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.484 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.484 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:39:48.484 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:39:48.484 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:48.494 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.535 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.854 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:39:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:39:48.855 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:39:49.288 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:49.288 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:39:49.288 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:39:49.288 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.289 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.289 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.289 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.289 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:39:49.289 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:39:49.629 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:49.668 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:39:49.668 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:39:49.668 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:39:49.668 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.668 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.669 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.669 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.669 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:39:49.669 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:39:49.723 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:49.723 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:39:49.723 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:39:49.724 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:39:49.724 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T13:39:49.724 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:39:49.724 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:49.782 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:49.782 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:49.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-09-13T13:39:49.817 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-09-13T13:39:49.817 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-09-13T13:39:49.817 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-09-13T13:39:49.817 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 82.el9 2024-09-13T13:39:49.817 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8274670 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-09-13T13:39:49.818 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:39:49.819 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:39:49.820 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:39:49.820 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:39:49.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:39:50.006 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:50.086 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:50.086 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:50.086 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:39:50.086 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:39:50.087 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:39:50.087 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-13T13:39:50.087 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:39:50.087 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:39:50.147 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:39:50.147 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:39:50.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 82.el9 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8274670 2024-09-13T13:39:50.191 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:39:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:39:50.193 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-09-13T13:39:50.193 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:39:50.193 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:39:50.193 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:39:50.194 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:39:50.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:39:50.481 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-09-13T13:39:50.864 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-09-13T13:39:51.871 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:39:51.895 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-09-13T13:39:51.897 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-13T13:39:51.913 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.I8kecQILnD 2024-09-13T13:39:51.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I8kecQILnD 2024-09-13T13:39:51.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.I8kecQILnD 2024-09-13T13:39:52.120 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.I8kecQILnD is 6KB 2024-09-13T13:39:52.168 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.I8kecQILnD 2024-09-13T13:39:52.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:39:52.186 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:39:52.216 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-09-13T13:39:52.218 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T13:39:52.233 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.lLentbOf0m 2024-09-13T13:39:52.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lLentbOf0m 2024-09-13T13:39:52.253 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:39:52.253 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:39:52.253 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:39:52.253 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:39:52.255 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:39:52.305 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.lLentbOf0m 2024-09-13T13:39:52.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-09-13T13:39:52.446 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.lLentbOf0m is 6KB 2024-09-13T13:39:52.467 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.lLentbOf0m 2024-09-13T13:39:52.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:39:52.544 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:39:52.544 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:39:52.544 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:39:52.544 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:39:52.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:39:52.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-09-13T13:40:11.079 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-13T13:40:11.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:40:11.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:17.185 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T13:40:17.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:40:17.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:22.417 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T13:40:22.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:40:22.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:22.701 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-13T13:40:22.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:40:22.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:35.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-13T13:40:40.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T13:40:41.153 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T13:40:44.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:40:44.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:45.761 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-09-13T13:40:47.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-13T13:40:49.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:40:49.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:50.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:40:50.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:52.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T13:40:53.249 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T13:40:54.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:40:54.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:59.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:40:59.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:02.653 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-13T13:41:04.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:41:04.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:05.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:41:05.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:13.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-09-13T13:41:17.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:41:17.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:20.733 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-13T13:41:23.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-09-13T13:41:23.649 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-09-13T13:41:25.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:41:25.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:25.952 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-09-13T13:41:26.459 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:41:26.460 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:41:26.460 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-09-13T13:41:26.474 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:41:26.475 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:41:26.475 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:41:26.475 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:41:27.475 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-13T13:41:27.476 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:41:27.546 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2024-09-13T13:41:27.575 DEBUG:teuthology.parallel:result is None 2024-09-13T13:41:38.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:41:38.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:38.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:41:38.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:38.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:41:38.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:41:38.762 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-13T13:41:39.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:41:39.081 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:41:39.081 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-13T13:41:39.095 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:41:39.096 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:41:39.096 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:41:39.096 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:41:39.252 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T13:41:39.956 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:41:39.956 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:41:39.956 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-13T13:41:39.971 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:41:39.971 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:41:39.971 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:41:39.971 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:41:40.097 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T13:41:40.097 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:41:40.217 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-09-13T13:41:40.242 DEBUG:teuthology.parallel:result is None 2024-09-13T13:41:40.971 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-13T13:41:40.972 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:41:41.035 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-09-13T13:41:41.062 DEBUG:teuthology.parallel:result is None 2024-09-13T13:41:54.428 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-13T13:41:54.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:41:54.809 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:41:54.809 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-13T13:41:54.826 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:41:54.827 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:41:54.827 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:41:54.827 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:41:55.828 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T13:41:55.828 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:41:55.900 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-09-13T13:41:55.929 DEBUG:teuthology.parallel:result is None 2024-09-13T13:41:55.930 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T13:41:55.940 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T13:41:55.940 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:41:55.943 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:41:55.947 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:41:55.950 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:41:55.971 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T13:41:55.979 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T13:41:55.985 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T13:41:55.985 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:41:56.003 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:41:56.006 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:41:56.008 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:41:56.088 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T13:41:56.094 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T13:41:56.094 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:41:56.097 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:41:56.099 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:41:56.102 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:41:56.129 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.136 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.139 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.141 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.147 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.151 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.161 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.173 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:41:56.175 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T13:41:56.181 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T13:41:56.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:41:56.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:41:56.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:41:56.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:41:56.223 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T13:41:56.232 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T13:41:56.233 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:41:56.257 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:41:56.261 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:41:56.263 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:41:56.280 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:56.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:56.402 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:56.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:56.529 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:41:56.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:41:56.614 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:56.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:56.716 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:56.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:56.847 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:41:56.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:41:56.916 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:56.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:57.025 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:57.079 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:57.115 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:41:57.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:41:57.183 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:57.244 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:41:57.291 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:57.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:41:57.377 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:41:57.377 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:41:57.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-09-13T13:41:57.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-09-13T13:41:57.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-09-13T13:41:57.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-09-13T13:41:57.492 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:41:57.504 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:41:57.519 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:41:57.526 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:41:58.009 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T13:41:58.022 INFO:teuthology.task.internal:Starting timer... 2024-09-13T13:41:58.022 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T13:41:58.033 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T13:41:58.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-09-13T13:41:58.136 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-09-13T13:41:58.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-09-13T13:41:58.188 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-09-13T13:41:58.190 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-09-13T13:41:58.240 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-09-13T13:41:58.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-09-13T13:41:58.299 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-09-13T13:41:58.301 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T13:41:58.301 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-09-13T13:41:58.335 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-09-13T13:41:58.336 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-09-13T13:41:58.366 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-09-13T13:41:58.366 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-09-13T13:41:58.393 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-09-13T13:41:58.393 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-09-13T13:41:58.419 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-09-13T13:41:58.419 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-09-13T13:41:58.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:41:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:41:58.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:41:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:41:58.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:41:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:41:58.507 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:41:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:41:58.533 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T13:41:58.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:41:58.567 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:41:58.599 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:41:58.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:41:58.667 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T13:41:58.677 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T13:41:58.688 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T13:41:58.689 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi100.front.sepia.ceph.com,smithi122.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T13:45:53.029 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-09-13T13:45:53.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:45:53.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:45:53.122 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-13T13:45:53.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-13T13:45:53.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:45:53.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:45:53.287 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-13T13:45:53.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-13T13:45:53.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:45:53.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:45:53.458 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-13T13:45:53.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-13T13:45:53.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:45:53.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:45:53.665 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-09-13T13:45:53.752 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-09-13T13:45:53.753 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T13:45:53.765 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T13:45:53.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:45:53.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:45:53.768 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:45:53.769 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:45:53.771 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:45:53.772 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:45:53.774 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:45:53.774 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:45:53.801 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:45:53.801 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:45:53.804 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:45:53.817 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:45:53.817 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:45:53.819 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:45:53.826 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:45:53.842 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-09-13T13:45:53.843 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:45:53.845 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-09-13T13:45:53.845 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-09-13T13:45:53.854 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-09-13T13:45:53.856 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-09-13T13:45:53.858 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-09-13T13:45:53.868 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:45:53.869 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-09-13T13:45:53.871 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:45:53.873 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:45:53.881 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-09-13T13:45:53.883 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:45:53.885 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:45:53.889 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:45:53.895 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:45:53.910 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:45:53.938 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:45:53.941 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:45:53.942 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-13T13:45:53.942 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.942 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.942 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.942 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.943 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.947 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.950 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:45:53.956 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:45:53.956 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-13T13:45:53.956 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.956 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.957 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.957 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.963 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:45:53.967 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:45:53.967 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-09-13T13:45:53.967 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.968 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.968 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.968 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:45:53.968 INFO:teuthology.run_tasks:Running task install... 2024-09-13T13:45:53.977 DEBUG:teuthology.task.install:project ceph 2024-09-13T13:45:53.977 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T13:45:53.977 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T13:45:53.977 INFO:teuthology.task.install:Using flavor: default 2024-09-13T13:45:53.982 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T13:45:53.983 INFO:teuthology.task.install:extra packages: [] 2024-09-13T13:45:53.983 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:45:53.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:45:53.986 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:45:53.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:45:53.987 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:45:53.987 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:45:53.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:45:53.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:45:54.147 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:45:54.147 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:45:54.159 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:45:54.159 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:45:54.176 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:45:54.176 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:45:54.178 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:45:54.178 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:45:54.317 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:45:54.317 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:45:54.317 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:45:54.319 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:45:54.320 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:45:54.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:45:54.335 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:45:54.335 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:45:54.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:45:54.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:45:54.339 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:45:54.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:45:54.345 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:45:54.345 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:45:54.348 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:45:54.348 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:45:54.362 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:45:54.362 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:45:54.365 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:45:54.365 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:45:54.411 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:45:54.416 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:45:54.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:45:54.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:45:54.486 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:45:54.494 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:45:54.505 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:45:54.508 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:45:54.551 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-09-13T13:45:54.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-09-13T13:45:54.565 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-09-13T13:45:54.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-09-13T13:45:54.570 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-09-13T13:45:54.572 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-09-13T13:45:54.589 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-09-13T13:45:54.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-09-13T13:45:54.793 INFO:teuthology.orchestra.run.smithi100.stdout:45 files removed 2024-09-13T13:45:54.815 INFO:teuthology.orchestra.run.smithi027.stdout:45 files removed 2024-09-13T13:45:54.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-09-13T13:45:54.821 INFO:teuthology.orchestra.run.smithi159.stdout:45 files removed 2024-09-13T13:45:54.824 INFO:teuthology.orchestra.run.smithi122.stdout:45 files removed 2024-09-13T13:45:54.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-09-13T13:45:54.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2024-09-13T13:45:54.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-09-13T13:45:55.436 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 286 kB/s | 84 kB 00:00 2024-09-13T13:45:55.452 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 264 kB/s | 84 kB 00:00 2024-09-13T13:45:55.464 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 254 kB/s | 84 kB 00:00 2024-09-13T13:45:55.467 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 230 kB/s | 84 kB 00:00 2024-09-13T13:45:55.720 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-13T13:45:55.727 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 104 kB/s | 24 kB 00:00 2024-09-13T13:45:55.757 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T13:45:55.760 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T13:45:56.004 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T13:45:56.011 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T13:45:56.041 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T13:45:56.043 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T13:45:57.027 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.2 MB 00:00 2024-09-13T13:45:57.046 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.2 MB 00:01 2024-09-13T13:45:57.140 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.2 MB 00:01 2024-09-13T13:45:57.927 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-13T13:45:59.064 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-13T13:46:00.296 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 20 MB 00:02 2024-09-13T13:46:02.336 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 20 MB 00:03 2024-09-13T13:46:02.792 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 20 MB 00:04 2024-09-13T13:46:04.790 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T13:46:06.047 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.5 MB 00:01 2024-09-13T13:46:06.207 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 123 kB/s | 19 kB 00:00 2024-09-13T13:46:06.646 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T13:46:07.143 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T13:46:07.628 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:46:07.855 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-13T13:46:08.546 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-13T13:46:08.661 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 19 kB 00:00 2024-09-13T13:46:09.118 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-13T13:46:09.953 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-13T13:46:10.140 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:46:13.293 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-09-13T13:46:14.507 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T13:46:15.391 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T13:46:15.554 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T13:46:15.755 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:46:15.763 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:46:15.764 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:46:15.765 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout:Install 30 Packages 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-09-13T13:46:15.766 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:15.767 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 62 M 2024-09-13T13:46:15.767 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:46:16.477 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-13T13:46:16.527 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 503 kB/s | 25 kB 00:00 2024-09-13T13:46:16.619 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-13T13:46:16.711 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 5.4 MB/s | 508 kB 00:00 2024-09-13T13:46:16.986 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 11 MB/s | 11 MB 00:00 2024-09-13T13:46:17.050 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:46:17.061 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:46:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:46:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:Install 30 Packages 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:17.064 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 62 M 2024-09-13T13:46:17.065 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:46:17.070 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 15 MB/s | 5.5 MB 00:00 2024-09-13T13:46:17.095 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): python3-ceph-argparse-19.3.0-4697.gd58f 416 kB/s | 45 kB 00:00 2024-09-13T13:46:17.121 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 2.8 MB/s | 143 kB 00:00 2024-09-13T13:46:17.146 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 3.1 MB/s | 161 kB 00:00 2024-09-13T13:46:17.171 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.3 MB/s | 321 kB 00:00 2024-09-13T13:46:17.196 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.9 MB/s | 305 kB 00:00 2024-09-13T13:46:17.221 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T13:46:17.430 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 15 MB/s | 22 MB 00:01 2024-09-13T13:46:17.588 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 273 kB/s | 107 kB 00:00 2024-09-13T13:46:17.614 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 247 kB/s | 45 kB 00:00 2024-09-13T13:46:17.639 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-09-13T13:46:17.689 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-13T13:46:17.731 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-13T13:46:17.750 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-13T13:46:17.769 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-09-13T13:46:17.773 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:46:17.774 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:46:17.775 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:46:17.776 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:Install 30 Packages 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 62 M 2024-09-13T13:46:17.777 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:46:17.789 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-13T13:46:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-13T13:46:17.817 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 378 kB/s | 25 kB 00:00 2024-09-13T13:46:17.832 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 356 kB/s | 36 kB 00:00 2024-09-13T13:46:17.849 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 799 kB/s | 47 kB 00:00 2024-09-13T13:46:17.908 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 611 kB/s | 26 kB 00:00 2024-09-13T13:46:17.925 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-13T13:46:17.933 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-13T13:46:17.950 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-09-13T13:46:17.956 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:46:17.974 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:46:17.975 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:46:17.976 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:46:17.977 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout:Install 30 Packages 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-09-13T13:46:17.978 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:17.979 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-13T13:46:17.979 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 62 M 2024-09-13T13:46:17.979 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:46:18.009 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T13:46:18.051 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 13 MB/s | 11 MB 00:00 2024-09-13T13:46:18.067 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-13T13:46:18.076 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 3.3 MB/s | 508 kB 00:00 2024-09-13T13:46:18.143 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 679 kB/s | 45 kB 00:00 2024-09-13T13:46:18.152 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-13T13:46:18.251 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-13T13:46:18.252 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 1.3 MB/s | 143 kB 00:00 2024-09-13T13:46:18.335 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-13T13:46:18.360 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 1.5 MB/s | 161 kB 00:00 2024-09-13T13:46:18.385 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-13T13:46:18.385 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.3 MB/s | 321 kB 00:00 2024-09-13T13:46:18.485 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 9.7 MB/s | 3.2 MB 00:00 2024-09-13T13:46:18.489 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:18.489 INFO:teuthology.orchestra.run.smithi027.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-13T13:46:18.590 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:46:18.594 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): ceph-common-19.3.0-4697.gd58f7306.el9. 16 MB/s | 22 MB 00:01 2024-09-13T13:46:18.619 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 1.2 MB/s | 305 kB 00:00 2024-09-13T13:46:18.630 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:46:18.630 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:46:18.644 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 384 kB/s | 99 kB 00:00 2024-09-13T13:46:18.745 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 711 kB/s | 107 kB 00:00 2024-09-13T13:46:18.770 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 360 kB/s | 45 kB 00:00 2024-09-13T13:46:18.796 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 909 kB/s | 160 kB 00:00 2024-09-13T13:46:18.821 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-13T13:46:18.846 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-13T13:46:18.872 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-13T13:46:18.897 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-13T13:46:18.914 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 534 kB/s | 36 kB 00:00 2024-09-13T13:46:18.931 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 804 kB/s | 47 kB 00:00 2024-09-13T13:46:18.989 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-13T13:46:19.006 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-13T13:46:19.023 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 337 kB/s | 26 kB 00:00 2024-09-13T13:46:19.049 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-13T13:46:19.082 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T13:46:19.133 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-13T13:46:19.201 INFO:teuthology.orchestra.run.smithi100.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 4.9 MB/s | 5.5 MB 00:01 2024-09-13T13:46:19.216 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-13T13:46:19.326 INFO:teuthology.orchestra.run.smithi100.stdout:(2/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 8.8 MB/s | 11 MB 00:01 2024-09-13T13:46:19.351 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:46:19.352 INFO:teuthology.orchestra.run.smithi100.stdout:(3/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 167 kB/s | 25 kB 00:00 2024-09-13T13:46:19.352 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:46:19.358 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-13T13:46:19.416 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-13T13:46:19.460 INFO:teuthology.orchestra.run.smithi100.stdout:(4/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 5.5 MB/s | 755 kB 00:00 2024-09-13T13:46:19.485 INFO:teuthology.orchestra.run.smithi100.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 3.7 MB/s | 508 kB 00:00 2024-09-13T13:46:19.533 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-09-13T13:46:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:19.538 INFO:teuthology.orchestra.run.smithi159.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-13T13:46:19.552 INFO:teuthology.orchestra.run.smithi100.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 676 kB/s | 45 kB 00:00 2024-09-13T13:46:19.572 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 4.4 MB/s | 5.5 MB 00:01 2024-09-13T13:46:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 1.9 MB/s | 143 kB 00:00 2024-09-13T13:46:19.634 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:46:19.670 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:46:19.670 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:46:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 1.9 MB/s | 161 kB 00:00 2024-09-13T13:46:19.756 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 7.7 MB/s | 11 MB 00:01 2024-09-13T13:46:19.781 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 121 kB/s | 25 kB 00:00 2024-09-13T13:46:19.840 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.9 MB/s | 755 kB 00:00 2024-09-13T13:46:19.898 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 4.2 MB/s | 508 kB 00:00 2024-09-13T13:46:19.937 INFO:teuthology.orchestra.run.smithi100.stdout:(9/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 12 MB/s | 22 MB 00:01 2024-09-13T13:46:19.965 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 680 kB/s | 45 kB 00:00 2024-09-13T13:46:19.970 INFO:teuthology.orchestra.run.smithi100.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 1.2 MB/s | 321 kB 00:00 2024-09-13T13:46:20.004 INFO:teuthology.orchestra.run.smithi100.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 4.5 MB/s | 305 kB 00:00 2024-09-13T13:46:20.032 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.1 MB/s | 143 kB 00:00 2024-09-13T13:46:20.087 INFO:teuthology.orchestra.run.smithi100.stdout:(12/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_ 8.8 MB/s | 5.5 MB 00:00 2024-09-13T13:46:20.091 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.7 MB/s | 161 kB 00:00 2024-09-13T13:46:20.113 INFO:teuthology.orchestra.run.smithi100.stdout:(13/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 699 kB/s | 99 kB 00:00 2024-09-13T13:46:20.175 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 16 MB/s | 5.5 MB 00:00 2024-09-13T13:46:20.208 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 2.7 MB/s | 321 kB 00:00 2024-09-13T13:46:20.233 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T13:46:20.259 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T13:46:20.305 INFO:teuthology.orchestra.run.smithi100.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-09-13T13:46:20.325 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:46:20.326 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:46:20.338 INFO:teuthology.orchestra.run.smithi100.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 201 kB/s | 45 kB 00:00 2024-09-13T13:46:20.397 INFO:teuthology.orchestra.run.smithi100.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 518 kB/s | 160 kB 00:00 2024-09-13T13:46:20.459 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 10 MB/s | 22 MB 00:02 2024-09-13T13:46:20.464 INFO:teuthology.orchestra.run.smithi100.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-09-13T13:46:20.522 INFO:teuthology.orchestra.run.smithi100.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-09-13T13:46:20.548 INFO:teuthology.orchestra.run.smithi100.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-09-13T13:46:20.582 INFO:teuthology.orchestra.run.smithi100.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 608 kB/s | 36 kB 00:00 2024-09-13T13:46:20.607 INFO:teuthology.orchestra.run.smithi100.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-13T13:46:20.609 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 302 kB/s | 45 kB 00:00 2024-09-13T13:46:20.632 INFO:teuthology.orchestra.run.smithi100.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 562 kB/s | 47 kB 00:00 2024-09-13T13:46:20.634 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 267 kB/s | 107 kB 00:00 2024-09-13T13:46:20.660 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 400 kB/s | 160 kB 00:00 2024-09-13T13:46:20.691 INFO:teuthology.orchestra.run.smithi100.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T13:46:20.716 INFO:teuthology.orchestra.run.smithi100.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 380 kB/s | 26 kB 00:00 2024-09-13T13:46:20.741 INFO:teuthology.orchestra.run.smithi100.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-13T13:46:20.766 INFO:teuthology.orchestra.run.smithi100.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T13:46:20.808 INFO:teuthology.orchestra.run.smithi100.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-13T13:46:20.810 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 281 kB/s | 42 kB 00:00 2024-09-13T13:46:20.834 INFO:teuthology.orchestra.run.smithi100.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-13T13:46:20.869 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 292 kB 00:00 2024-09-13T13:46:20.901 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:46:20.902 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-13T13:46:20.927 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-13T13:46:20.934 INFO:teuthology.orchestra.run.smithi100.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-13T13:46:20.953 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-13T13:46:20.978 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-13T13:46:20.988 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:46:21.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:46:21.037 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 601 kB/s | 26 kB 00:00 2024-09-13T13:46:21.040 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:46:21.062 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-13T13:46:21.063 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:21.087 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-13T13:46:21.089 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:21.093 INFO:teuthology.orchestra.run.smithi100.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-09-13T13:46:21.112 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T13:46:21.163 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-13T13:46:21.185 INFO:teuthology.orchestra.run.smithi100.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-13T13:46:21.197 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-13T13:46:21.252 INFO:teuthology.orchestra.run.smithi100.stdout:(32/32): librados2-19.3.0-4697.gd58f7306.el9.x8 8.3 MB/s | 3.5 MB 00:00 2024-09-13T13:46:21.257 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:21.257 INFO:teuthology.orchestra.run.smithi100.stdout:Total 18 MB/s | 62 MB 00:03 2024-09-13T13:46:21.283 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:21.314 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-13T13:46:21.337 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:21.361 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:46:21.397 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:46:21.397 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:46:21.472 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-09-13T13:46:21.573 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-13T13:46:21.656 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): librados2-19.3.0-4697.gd58f7306.el9.x8 7.5 MB/s | 3.5 MB 00:00 2024-09-13T13:46:21.662 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:21.662 INFO:teuthology.orchestra.run.smithi122.stdout:Total 17 MB/s | 62 MB 00:03 2024-09-13T13:46:21.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:46:21.768 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:46:21.792 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:21.798 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:46:21.803 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:46:21.803 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:46:21.885 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:21.914 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:46:21.946 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:46:21.975 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:46:21.988 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:46:22.001 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:22.011 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:22.038 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:22.064 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:22.069 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:46:22.069 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:46:22.093 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:46:22.145 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:46:22.216 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:22.239 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:22.245 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:22.292 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:22.301 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:22.335 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:46:22.448 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:46:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:46:22.491 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:46:22.492 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:46:22.535 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:46:22.576 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:46:22.606 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:46:22.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:46:22.647 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:46:22.765 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:22.858 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:22.945 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:46:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:22.984 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:46:23.026 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:46:23.110 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:46:23.118 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:23.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:46:23.180 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:23.193 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:46:23.202 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:23.266 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:23.292 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:46:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:46:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:46:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:46:23.517 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:46:23.546 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:46:23.586 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:46:23.586 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:46:23.675 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:46:23.707 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:46:23.712 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:46:23.732 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:46:23.737 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:23.754 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:23.777 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:23.889 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:46:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:46:23.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:46:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:23.982 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:24.012 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:46:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:24.040 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:24.071 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:46:24.097 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:46:24.107 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:46:24.128 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:46:24.153 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:46:24.184 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:24.207 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:46:24.403 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:24.428 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:46:24.465 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:46:24.492 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:24.584 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:24.599 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:46:24.627 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:24.671 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:46:24.682 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:24.701 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:24.760 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:46:24.764 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:24.788 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:46:24.793 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:46:24.837 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:46:24.873 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:46:24.914 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:24.938 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:24.945 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:25.026 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:25.031 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:46:25.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:46:25.148 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:25.177 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:46:25.194 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:46:25.211 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:46:25.231 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:46:25.263 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:46:25.275 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:46:25.293 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:46:25.322 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:46:25.344 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:25.355 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:46:25.367 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:25.431 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:46:25.465 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:46:25.583 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:46:25.617 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:46:25.657 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:46:25.676 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:46:25.706 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:46:25.718 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:25.736 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:46:25.776 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:46:25.801 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:25.838 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:46:25.884 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:46:26.096 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:46:26.139 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:26.231 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:46:26.261 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:46:26.305 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:46:26.373 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:46:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:26.415 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:26.452 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:46:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:46:26.797 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:46:26.825 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:26.897 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:46:26.976 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:46:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:46:27.032 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:27.069 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:27.084 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:27.105 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:27.193 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:27.193 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:46:27.193 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:46:27.193 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:27.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:27.664 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:27.696 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:27.790 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:27.790 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:46:27.790 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:46:27.790 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:27.823 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:28.809 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:28.852 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:29.439 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:29.489 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:29.585 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:29.585 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:46:29.585 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:46:29.585 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:29.615 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:30.094 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:46:30.144 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:30.240 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:46:30.241 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:46:30.241 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:46:30.241 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:30.270 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-09-13T13:46:37.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-09-13T13:46:37.982 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:38.967 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:38.994 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:38.996 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:39.007 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:39.035 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:39.035 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:46:39.035 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:46:39.035 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:39.035 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:39.036 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:39.037 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:39.086 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:39.086 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:39.135 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:39.135 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:39.693 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-09-13T13:46:39.694 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-09-13T13:46:40.254 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-09-13T13:46:40.255 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:40.262 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:40.262 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:46:40.262 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:46:40.262 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:46:40.263 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:46:40.264 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:46:40.265 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:46:40.266 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:46:40.266 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:40.266 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:46:40.266 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:46:40.306 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:46:40.307 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:46:40.308 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:46:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:46:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:46:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:46:40.309 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:46:40.310 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:46:40.666 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:40.692 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:40.694 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:41.238 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:46:41.262 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:41.264 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:46:41.319 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:46:41.320 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:46:41.321 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:46:41.322 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:46:41.322 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:46:41.322 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:46:41.322 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:41.322 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:46:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:46:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.485 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:46:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:41.488 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:46:41.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-09-13T13:46:41.872 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2024-09-13T13:46:42.036 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:42.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:46:42.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:46:42.036 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:46:42.037 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:46:42.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:46:42.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:46:42.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:46:42.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:42.039 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:46:42.040 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:46:42.234 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:46:42.371 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:46:42.371 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:42.372 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:42.373 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 51 M 2024-09-13T13:46:42.374 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 216 M 2024-09-13T13:46:42.374 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:46:42.439 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:46:42.572 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:46:42.572 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:42.573 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-09-13T13:46:42.574 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:42.574 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 51 M 2024-09-13T13:46:42.574 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 216 M 2024-09-13T13:46:42.574 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:46:42.606 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:46:42.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:46:42.608 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:46:42.608 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:46:42.608 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:46:42.608 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:46:42.608 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:46:42.609 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:46:42.758 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 761 kB/s | 161 kB 00:00 2024-09-13T13:46:42.816 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-13T13:46:42.925 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 382 kB/s | 64 kB 00:00 2024-09-13T13:46:42.999 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T13:46:43.024 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 695 kB/s | 161 kB 00:00 2024-09-13T13:46:43.075 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 849 kB/s | 64 kB 00:00 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.127 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:46:43.128 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:46:43.129 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:46:43.130 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:46:43.130 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:43.130 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:46:43.474 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-09-13T13:46:43.648 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:46:43.649 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.650 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:43.651 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:46:44.026 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-09-13T13:46:44.050 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:46:44.183 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:46:44.183 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:44.183 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-13T13:46:44.184 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:44.185 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 51 M 2024-09-13T13:46:44.185 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 216 M 2024-09-13T13:46:44.185 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:46:44.367 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 28 MB/s | 50 MB 00:01 2024-09-13T13:46:44.369 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:44.369 INFO:teuthology.orchestra.run.smithi027.stdout:Total 25 MB/s | 51 MB 00:01 2024-09-13T13:46:44.398 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:46:44.420 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:46:44.420 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:46:44.588 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:46:44.715 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 587 kB/s | 161 kB 00:00 2024-09-13T13:46:44.720 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:46:44.720 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:44.721 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-13T13:46:44.722 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:44.722 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 51 M 2024-09-13T13:46:44.722 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 216 M 2024-09-13T13:46:44.722 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:46:44.821 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:46:44.821 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:46:44.896 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-13T13:46:44.937 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T13:46:45.025 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 22 MB/s | 50 MB 00:02 2024-09-13T13:46:45.027 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:45.027 INFO:teuthology.orchestra.run.smithi159.stdout:Total 21 MB/s | 51 MB 00:02 2024-09-13T13:46:45.055 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:46:45.071 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 495 kB/s | 161 kB 00:00 2024-09-13T13:46:45.075 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:46:45.076 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:46:45.207 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 397 kB/s | 304 kB 00:00 2024-09-13T13:46:45.324 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 105 kB/s | 64 kB 00:00 2024-09-13T13:46:45.383 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:46:45.456 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:46:45.456 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:46:45.469 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:46:45.518 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:46:45.543 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:45.988 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:46:46.074 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-13T13:46:46.076 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:46.077 INFO:teuthology.orchestra.run.smithi100.stdout:Total 27 MB/s | 51 MB 00:01 2024-09-13T13:46:46.079 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:46:46.104 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:46:46.125 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:46:46.125 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:46:46.127 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:46:46.151 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:46.504 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:46:46.505 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:46:47.038 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:46:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:46:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:46:47.192 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:47.529 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 18 MB/s | 50 MB 00:02 2024-09-13T13:46:47.531 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:47.532 INFO:teuthology.orchestra.run.smithi122.stdout:Total 18 MB/s | 51 MB 00:02 2024-09-13T13:46:47.559 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:46:47.579 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:46:47.579 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:46:47.959 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:46:47.960 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:46:48.494 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:46:48.586 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:46:48.635 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:46:48.665 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:50.584 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:50.661 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:50.952 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:51.026 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:51.924 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:52.007 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:53.086 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:53.086 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:46:53.086 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:46:53.086 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:46:53.344 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:53.344 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:46:53.344 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:46:53.344 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:46:53.405 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:53.448 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:46:53.480 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:53.671 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:46:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:53.672 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:46:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:53.672 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:46:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:46:53.673 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:53.673 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:46:53.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-09-13T13:46:53.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2024-09-13T13:46:54.328 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:46:54.357 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:54.357 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:46:54.357 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:46:54.358 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:46:54.472 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:46:54.475 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:46:54.476 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:46:54.477 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:46:54.478 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout:Install 37 Packages 2024-09-13T13:46:54.479 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:46:54.480 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 31 M 2024-09-13T13:46:54.480 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 109 M 2024-09-13T13:46:54.480 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:46:54.500 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:46:54.644 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:46:54.646 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:46:54.647 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:46:54.648 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:46:54.649 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:Install 37 Packages 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 31 M 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 109 M 2024-09-13T13:46:54.650 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:46:54.676 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:46:54.676 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:54.676 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:54.677 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:46:54.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-09-13T13:46:55.234 INFO:teuthology.orchestra.run.smithi159.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-13T13:46:55.393 INFO:teuthology.orchestra.run.smithi159.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-13T13:46:55.417 INFO:teuthology.orchestra.run.smithi027.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-13T13:46:55.468 INFO:teuthology.orchestra.run.smithi159.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-13T13:46:55.501 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:46:55.568 INFO:teuthology.orchestra.run.smithi159.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.8 MB/s | 291 kB 00:00 2024-09-13T13:46:55.635 INFO:teuthology.orchestra.run.smithi159.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-13T13:46:55.653 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:46:55.656 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:46:55.657 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:46:55.658 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout:Install 37 Packages 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 M 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 109 M 2024-09-13T13:46:55.660 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:46:55.835 INFO:teuthology.orchestra.run.smithi159.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 950 kB/s | 253 kB 00:00 2024-09-13T13:46:55.976 INFO:teuthology.orchestra.run.smithi027.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-09-13T13:46:56.028 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:46:56.028 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:46:56.028 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:46:56.028 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:46:56.028 INFO:teuthology.orchestra.run.smithi159.stdout:(7/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-13T13:46:56.053 INFO:teuthology.orchestra.run.smithi159.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-13T13:46:56.101 INFO:teuthology.orchestra.run.smithi027.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 2.9 MB/s | 2.4 MB 00:00 2024-09-13T13:46:56.103 INFO:teuthology.orchestra.run.smithi159.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-13T13:46:56.145 INFO:teuthology.orchestra.run.smithi159.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-09-13T13:46:56.168 INFO:teuthology.orchestra.run.smithi027.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 4.3 MB/s | 291 kB 00:00 2024-09-13T13:46:56.170 INFO:teuthology.orchestra.run.smithi159.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T13:46:56.229 INFO:teuthology.orchestra.run.smithi159.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-13T13:46:56.279 INFO:teuthology.orchestra.run.smithi159.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-13T13:46:56.296 INFO:teuthology.orchestra.run.smithi159.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 518 kB/s | 35 kB 00:00 2024-09-13T13:46:56.325 INFO:teuthology.orchestra.run.smithi159.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-13T13:46:56.350 INFO:teuthology.orchestra.run.smithi159.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-13T13:46:56.359 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:56.360 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:46:56.375 INFO:teuthology.orchestra.run.smithi159.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 649 kB/s | 51 kB 00:00 2024-09-13T13:46:56.400 INFO:teuthology.orchestra.run.smithi159.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-13T13:46:56.417 INFO:teuthology.orchestra.run.smithi100.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-13T13:46:56.435 INFO:teuthology.orchestra.run.smithi027.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 4.7 MB/s | 4.8 MB 00:01 2024-09-13T13:46:56.459 INFO:teuthology.orchestra.run.smithi159.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 748 kB/s | 43 kB 00:00 2024-09-13T13:46:56.484 INFO:teuthology.orchestra.run.smithi159.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-13T13:46:56.518 INFO:teuthology.orchestra.run.smithi159.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-09-13T13:46:56.543 INFO:teuthology.orchestra.run.smithi159.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-13T13:46:56.575 INFO:teuthology.orchestra.run.smithi100.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-13T13:46:56.577 INFO:teuthology.orchestra.run.smithi159.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 672 kB/s | 18 kB 00:00 2024-09-13T13:46:56.610 INFO:teuthology.orchestra.run.smithi159.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-13T13:46:56.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-09-13T13:46:56.634 INFO:teuthology.orchestra.run.smithi100.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-13T13:46:56.636 INFO:teuthology.orchestra.run.smithi159.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-09-13T13:46:56.661 INFO:teuthology.orchestra.run.smithi159.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 281 kB/s | 23 kB 00:00 2024-09-13T13:46:56.686 INFO:teuthology.orchestra.run.smithi159.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-13T13:46:56.709 INFO:teuthology.orchestra.run.smithi100.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 3.8 MB/s | 291 kB 00:00 2024-09-13T13:46:56.712 INFO:teuthology.orchestra.run.smithi159.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-13T13:46:56.737 INFO:teuthology.orchestra.run.smithi159.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-09-13T13:46:56.762 INFO:teuthology.orchestra.run.smithi159.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T13:46:56.787 INFO:teuthology.orchestra.run.smithi159.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-13T13:46:56.813 INFO:teuthology.orchestra.run.smithi159.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T13:46:56.838 INFO:teuthology.orchestra.run.smithi159.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-13T13:46:56.863 INFO:teuthology.orchestra.run.smithi159.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T13:46:56.889 INFO:teuthology.orchestra.run.smithi159.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T13:46:56.914 INFO:teuthology.orchestra.run.smithi159.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-13T13:46:56.939 INFO:teuthology.orchestra.run.smithi159.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T13:46:56.943 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:56.943 INFO:teuthology.orchestra.run.smithi159.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-13T13:46:56.994 INFO:teuthology.orchestra.run.smithi027.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 307 kB/s | 253 kB 00:00 2024-09-13T13:46:57.095 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:46:57.133 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:46:57.134 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:46:57.194 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:42 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:46:57.201 INFO:teuthology.orchestra.run.smithi100.stdout:(5/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-13T13:46:57.226 INFO:teuthology.orchestra.run.smithi100.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 490 kB/s | 253 kB 00:00 2024-09-13T13:46:57.302 INFO:teuthology.orchestra.run.smithi100.stdout:(7/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 5.4 MB/s | 4.8 MB 00:00 2024-09-13T13:46:57.329 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:46:57.331 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:57.331 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:46:57.331 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:57.331 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:46:57.331 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:46:57.332 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:46:57.333 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:46:57.334 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout:Install 37 Packages 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:46:57.335 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 31 M 2024-09-13T13:46:57.336 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 109 M 2024-09-13T13:46:57.336 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:46:57.344 INFO:teuthology.orchestra.run.smithi027.stdout:(7/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 13 MB/s | 17 MB 00:01 2024-09-13T13:46:57.369 INFO:teuthology.orchestra.run.smithi100.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 750 kB/s | 106 kB 00:00 2024-09-13T13:46:57.494 INFO:teuthology.orchestra.run.smithi027.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:01 2024-09-13T13:46:57.511 INFO:teuthology.orchestra.run.smithi100.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 882 kB/s | 125 kB 00:00 2024-09-13T13:46:57.603 INFO:teuthology.orchestra.run.smithi100.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 450 kB/s | 135 kB 00:00 2024-09-13T13:46:57.626 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:46:57.626 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:46:57.686 INFO:teuthology.orchestra.run.smithi100.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.2 MB/s | 215 kB 00:00 2024-09-13T13:46:57.753 INFO:teuthology.orchestra.run.smithi100.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-13T13:46:57.795 INFO:teuthology.orchestra.run.smithi100.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 840 kB/s | 35 kB 00:00 2024-09-13T13:46:57.821 INFO:teuthology.orchestra.run.smithi100.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T13:46:57.828 INFO:teuthology.orchestra.run.smithi027.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 128 kB/s | 106 kB 00:00 2024-09-13T13:46:57.846 INFO:teuthology.orchestra.run.smithi100.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-13T13:46:57.863 INFO:teuthology.orchestra.run.smithi100.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-13T13:46:57.880 INFO:teuthology.orchestra.run.smithi100.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-13T13:46:57.947 INFO:teuthology.orchestra.run.smithi100.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-13T13:46:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 221 kB/s | 22 kB 00:00 2024-09-13T13:46:57.981 INFO:teuthology.orchestra.run.smithi100.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-13T13:46:58.040 INFO:teuthology.orchestra.run.smithi100.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-13T13:46:58.045 INFO:teuthology.orchestra.run.smithi027.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-13T13:46:58.065 INFO:teuthology.orchestra.run.smithi100.stdout:(22/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.6 MB/s | 358 kB 00:00 2024-09-13T13:46:58.090 INFO:teuthology.orchestra.run.smithi100.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-13T13:46:58.132 INFO:teuthology.orchestra.run.smithi100.stdout:(24/37): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-13T13:46:58.157 INFO:teuthology.orchestra.run.smithi100.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-09-13T13:46:58.182 INFO:teuthology.orchestra.run.smithi100.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-09-13T13:46:58.186 INFO:teuthology.orchestra.run.smithi027.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 180 kB/s | 125 kB 00:00 2024-09-13T13:46:58.208 INFO:teuthology.orchestra.run.smithi100.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-13T13:46:58.239 INFO:teuthology.orchestra.run.smithi122.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-13T13:46:58.241 INFO:teuthology.orchestra.run.smithi100.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-13T13:46:58.267 INFO:teuthology.orchestra.run.smithi100.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-13T13:46:58.292 INFO:teuthology.orchestra.run.smithi100.stdout:(30/37): python3-jaraco-text-3.2.0-6.el9.noarch 146 kB/s | 20 kB 00:00 2024-09-13T13:46:58.317 INFO:teuthology.orchestra.run.smithi100.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-13T13:46:58.343 INFO:teuthology.orchestra.run.smithi100.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T13:46:58.368 INFO:teuthology.orchestra.run.smithi100.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-13T13:46:58.393 INFO:teuthology.orchestra.run.smithi100.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T13:46:58.414 INFO:teuthology.orchestra.run.smithi122.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-13T13:46:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-13T13:46:58.452 INFO:teuthology.orchestra.run.smithi100.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-09-13T13:46:58.481 INFO:teuthology.orchestra.run.smithi122.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-13T13:46:58.485 INFO:teuthology.orchestra.run.smithi100.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-13T13:46:58.490 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:58.490 INFO:teuthology.orchestra.run.smithi100.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-13T13:46:58.578 INFO:teuthology.orchestra.run.smithi027.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 286 kB/s | 215 kB 00:00 2024-09-13T13:46:58.582 INFO:teuthology.orchestra.run.smithi122.stdout:(4/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 14 MB/s | 4.8 MB 00:00 2024-09-13T13:46:58.603 INFO:teuthology.orchestra.run.smithi027.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 342 kB/s | 188 kB 00:00 2024-09-13T13:46:58.615 INFO:teuthology.orchestra.run.smithi122.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.1 MB/s | 291 kB 00:00 2024-09-13T13:46:58.640 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:46:58.679 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:46:58.679 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:46:58.721 INFO:teuthology.orchestra.run.smithi027.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 661 kB/s | 77 kB 00:00 2024-09-13T13:46:58.746 INFO:teuthology.orchestra.run.smithi027.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 308 kB/s | 172 kB 00:00 2024-09-13T13:46:58.829 INFO:teuthology.orchestra.run.smithi027.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 470 kB/s | 51 kB 00:00 2024-09-13T13:46:58.855 INFO:teuthology.orchestra.run.smithi027.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-13T13:46:58.880 INFO:teuthology.orchestra.run.smithi027.stdout:(18/37): python3-markupsafe-1.1.1-12.el9.x86_64 115 kB/s | 35 kB 00:00 2024-09-13T13:46:58.922 INFO:teuthology.orchestra.run.smithi027.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-09-13T13:46:58.964 INFO:teuthology.orchestra.run.smithi027.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-09-13T13:46:58.985 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:46:58.989 INFO:teuthology.orchestra.run.smithi027.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-13T13:46:58.991 INFO:teuthology.orchestra.run.smithi122.stdout:(6/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-13T13:46:59.015 INFO:teuthology.orchestra.run.smithi027.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-13T13:46:59.040 INFO:teuthology.orchestra.run.smithi027.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-13T13:46:59.058 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:46:59.065 INFO:teuthology.orchestra.run.smithi027.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-13T13:46:59.090 INFO:teuthology.orchestra.run.smithi027.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-13T13:46:59.099 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:46:59.116 INFO:teuthology.orchestra.run.smithi027.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-13T13:46:59.127 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:46:59.141 INFO:teuthology.orchestra.run.smithi027.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-13T13:46:59.166 INFO:teuthology.orchestra.run.smithi027.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-13T13:46:59.173 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:46:59.174 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:46:59.191 INFO:teuthology.orchestra.run.smithi027.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-13T13:46:59.198 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:46:59.217 INFO:teuthology.orchestra.run.smithi027.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T13:46:59.242 INFO:teuthology.orchestra.run.smithi027.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-13T13:46:59.249 INFO:teuthology.orchestra.run.smithi122.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 380 kB/s | 253 kB 00:00 2024-09-13T13:46:59.260 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:46:59.267 INFO:teuthology.orchestra.run.smithi027.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-13T13:46:59.292 INFO:teuthology.orchestra.run.smithi027.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T13:46:59.293 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:46:59.318 INFO:teuthology.orchestra.run.smithi027.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-13T13:46:59.343 INFO:teuthology.orchestra.run.smithi027.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-13T13:46:59.366 INFO:teuthology.orchestra.run.smithi122.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-13T13:46:59.368 INFO:teuthology.orchestra.run.smithi027.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T13:46:59.385 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:46:59.393 INFO:teuthology.orchestra.run.smithi027.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T13:46:59.399 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:46:59.399 INFO:teuthology.orchestra.run.smithi027.stdout:Total 6.2 MB/s | 31 MB 00:04 2024-09-13T13:46:59.458 INFO:teuthology.orchestra.run.smithi122.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 228 kB/s | 106 kB 00:00 2024-09-13T13:46:59.463 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:46:59.484 INFO:teuthology.orchestra.run.smithi122.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-13T13:46:59.512 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:46:59.547 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:46:59.559 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:46:59.577 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:46:59.603 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:46:59.603 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:46:59.611 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:46:59.645 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:46:59.659 INFO:teuthology.orchestra.run.smithi122.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-13T13:46:59.692 INFO:teuthology.orchestra.run.smithi122.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2024-09-13T13:46:59.694 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:46:59.718 INFO:teuthology.orchestra.run.smithi122.stdout:(13/37): python3-urllib3-1.26.5-6.el9.noarch.rp 828 kB/s | 215 kB 00:00 2024-09-13T13:46:59.743 INFO:teuthology.orchestra.run.smithi122.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-13T13:46:59.746 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:46:59.780 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:46:59.788 INFO:teuthology.orchestra.run.smithi122.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 51 kB 00:00 2024-09-13T13:46:59.813 INFO:teuthology.orchestra.run.smithi122.stdout:(16/37): python3-markupsafe-1.1.1-12.el9.x86_64 288 kB/s | 35 kB 00:00 2024-09-13T13:46:59.828 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:46:59.839 INFO:teuthology.orchestra.run.smithi122.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-09-13T13:46:59.864 INFO:teuthology.orchestra.run.smithi122.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 529 kB/s | 77 kB 00:00 2024-09-13T13:46:59.871 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:46:59.914 INFO:teuthology.orchestra.run.smithi122.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T13:46:59.941 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:46:59.948 INFO:teuthology.orchestra.run.smithi122.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-09-13T13:46:59.973 INFO:teuthology.orchestra.run.smithi122.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-09-13T13:47:00.007 INFO:teuthology.orchestra.run.smithi122.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-13T13:47:00.041 INFO:teuthology.orchestra.run.smithi122.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 693 kB/s | 18 kB 00:00 2024-09-13T13:47:00.066 INFO:teuthology.orchestra.run.smithi122.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-13T13:47:00.089 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:47:00.090 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:00.091 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:00.091 INFO:teuthology.orchestra.run.smithi122.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-09-13T13:47:00.117 INFO:teuthology.orchestra.run.smithi122.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 314 kB/s | 23 kB 00:00 2024-09-13T13:47:00.142 INFO:teuthology.orchestra.run.smithi122.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-13T13:47:00.167 INFO:teuthology.orchestra.run.smithi122.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-13T13:47:00.170 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:47:00.192 INFO:teuthology.orchestra.run.smithi122.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-13T13:47:00.223 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:47:00.243 INFO:teuthology.orchestra.run.smithi122.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-09-13T13:47:00.262 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:47:00.268 INFO:teuthology.orchestra.run.smithi122.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-09-13T13:47:00.301 INFO:teuthology.orchestra.run.smithi122.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-13T13:47:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:47:00.327 INFO:teuthology.orchestra.run.smithi122.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-09-13T13:47:00.353 INFO:teuthology.orchestra.run.smithi122.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-09-13T13:47:00.375 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:47:00.377 INFO:teuthology.orchestra.run.smithi122.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T13:47:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-09-13T13:47:00.427 INFO:teuthology.orchestra.run.smithi122.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T13:47:00.432 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:00.432 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.9 MB/s | 31 MB 00:03 2024-09-13T13:47:00.512 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:00.561 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:47:00.572 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:47:00.585 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:00.604 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:47:00.615 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:47:00.624 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:00.624 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:00.641 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:47:00.693 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:47:00.721 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:47:00.783 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:47:00.793 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:47:00.823 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:47:00.834 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:47:00.926 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:47:00.951 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:47:01.012 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:47:01.054 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:47:01.098 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:47:01.105 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:01.106 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:01.112 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:47:01.128 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:47:01.142 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:01.162 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:47:01.167 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:01.167 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:01.167 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:47:01.167 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:01.167 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:01.168 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:01.196 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:47:01.236 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:47:01.279 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:47:01.333 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:47:01.379 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:47:01.417 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:47:01.494 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:47:01.633 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:01.634 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:47:01.696 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:47:01.713 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:47:01.733 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:47:01.765 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:47:01.769 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:47:01.812 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:47:01.836 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:47:01.870 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:47:01.885 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:47:01.917 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:47:01.934 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:47:02.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:47:02.100 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:47:02.127 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:47:02.146 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:47:02.180 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:47:02.186 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:47:02.210 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:47:02.245 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:47:02.271 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:47:02.278 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:47:02.319 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:47:02.360 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:47:02.363 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:47:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:47:02.426 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:47:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:47:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:47:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:47:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:47:02.562 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:47:02.571 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:47:02.583 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:47:02.643 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:02.655 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:47:02.669 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:02.669 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:02.669 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:47:02.669 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:02.670 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:02.670 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:02.723 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:47:02.728 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:47:02.736 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:47:02.764 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:02.773 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:47:02.790 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:02.790 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:02.790 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:47:02.790 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:02.791 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:02.791 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:02.814 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:47:02.865 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:47:02.865 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:47:02.902 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:47:02.944 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:47:02.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:47:02.984 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:47:03.025 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:47:03.028 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:47:03.066 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:47:03.100 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:47:03.134 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:47:03.135 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:03.159 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:03.175 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:47:03.218 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:47:03.222 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:47:03.252 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:47:03.274 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:47:03.300 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:47:03.341 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:47:03.355 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:47:03.412 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:03.422 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:47:03.434 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:03.434 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:03.434 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:47:03.435 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:03.435 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:03.435 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:03.445 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:47:03.481 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:47:03.499 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:03.570 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:47:03.620 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:47:03.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:47:03.704 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:47:03.742 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:47:03.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:47:03.794 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:47:03.824 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:03.851 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:03.855 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:47:04.041 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:47:04.097 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:47:04.184 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:47:04.272 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:47:04.273 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:04.300 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:04.300 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:04.301 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:47:04.301 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:04.301 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:04.301 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:04.312 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:47:04.441 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:47:04.602 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:47:04.640 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:04.665 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:47:04.665 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:04.665 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:47:04.666 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:04.666 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:47:04.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:04.748 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:04.772 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:04.772 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:04.772 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:47:04.772 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:04.773 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:04.773 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:04.991 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:05.015 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:05.159 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:05.159 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:47:05.159 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:47:05.160 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:47:05.161 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:47:05.162 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:47:05.162 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:47:05.163 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:47:05.409 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:05.438 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:05.917 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:05.943 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:06.162 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:47:06.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:47:06.163 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:47:06.164 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:47:06.165 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:06.166 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:06.176 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:06.213 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:06.213 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:06.213 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:06.213 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:47:06.213 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:06.214 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:06.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:06.222 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:47:06.222 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:06.222 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:47:06.222 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:06.223 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:47:06.223 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:06.269 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:06.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2024-09-13T13:47:06.688 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:47:06.710 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:06.710 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:47:06.710 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:06.710 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:47:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:06.823 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:06.823 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:47:06.823 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:47:06.823 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:47:06.824 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:47:06.825 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:47:06.826 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:47:06.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:47:06.937 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:47:06.961 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:07.027 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:07.067 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:07.159 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:07.200 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:07.202 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:07.202 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:07.259 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2024-09-13T13:47:07.825 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:07.827 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:47:07.828 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:47:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:07.830 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:07.960 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:07.961 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 783 k 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 789 k 2024-09-13T13:47:07.962 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:47:08.051 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:47:08.052 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:47:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:47:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:47:08.053 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:47:08.054 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:47:08.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:47:08.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:47:08.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:47:08.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:47:08.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:47:08.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-09-13T13:47:08.218 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-13T13:47:08.219 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:08.219 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.0 MB/s | 783 kB 00:00 2024-09-13T13:47:08.219 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:08.225 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:08.225 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:08.231 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:08.231 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:08.307 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:08.698 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:08.766 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:08.781 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:47:08.782 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:47:08.783 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:47:08.784 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:47:08.784 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:47:08.784 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:08.785 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:47:08.785 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:47:08.785 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:47:08.785 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:47:08.786 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:47:08.786 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:47:08.786 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:47:08.786 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:47:08.786 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:47:08.856 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:08.898 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:08.899 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:47:08.899 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:08.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-09-13T13:47:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:47:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:47:09.056 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:47:09.057 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:09.058 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:09.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:09.462 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:09.518 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:09.650 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:09.651 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 783 k 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 789 k 2024-09-13T13:47:09.652 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:09.654 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:09.708 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:47:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:47:09.710 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:09.711 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:09.931 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-13T13:47:09.932 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:09.932 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.7 MB/s | 783 kB 00:00 2024-09-13T13:47:09.932 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:09.937 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:09.937 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:09.943 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:09.944 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:09.970 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:10.016 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:10.066 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:10.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-09-13T13:47:10.111 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:10.112 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:47:10.112 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:10.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-09-13T13:47:10.228 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:10.365 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:10.366 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 145 k 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 432 k 2024-09-13T13:47:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:10.461 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:10.569 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:10.577 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 690 kB/s | 145 kB 00:00 2024-09-13T13:47:10.578 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:10.578 INFO:teuthology.orchestra.run.smithi159.stdout:Total 684 kB/s | 145 kB 00:00 2024-09-13T13:47:10.578 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:10.586 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:10.586 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:10.644 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:10.644 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:10.667 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:10.756 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:10.774 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:10.784 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:10.797 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:10.799 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:10.799 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:10.862 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:10.863 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-09-13T13:47:10.887 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:10.887 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:10.887 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:47:10.887 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:10.912 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:10.913 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 783 k 2024-09-13T13:47:10.914 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 789 k 2024-09-13T13:47:10.914 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:10.945 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:11.201 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-13T13:47:11.201 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:11.202 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.6 MB/s | 783 kB 00:00 2024-09-13T13:47:11.203 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:11.208 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:11.208 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:11.214 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:11.214 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:11.287 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:11.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:47:11.425 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:11.558 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:11.559 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 783 k 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 789 k 2024-09-13T13:47:11.560 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:11.773 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:11.803 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:11.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:11.804 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:11.804 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:11.804 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:11.804 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:11.822 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T13:47:11.822 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:11.823 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T13:47:11.823 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:11.828 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:11.828 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:11.834 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:11.834 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:11.856 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:11.904 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:11.984 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:12.013 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2024-09-13T13:47:12.119 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:12.120 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:47:12.121 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:12.121 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 145 k 2024-09-13T13:47:12.121 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 432 k 2024-09-13T13:47:12.121 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:12.210 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:12.314 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 752 kB/s | 145 kB 00:00 2024-09-13T13:47:12.314 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:12.315 INFO:teuthology.orchestra.run.smithi100.stdout:Total 745 kB/s | 145 kB 00:00 2024-09-13T13:47:12.315 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:12.323 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:12.323 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:12.375 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:12.380 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:12.381 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:12.441 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:12.442 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:12.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:12.442 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:12.483 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:12.596 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:12.615 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:12.640 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:12.640 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:12.641 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:47:12.641 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:12.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:47:12.689 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:12.730 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:12.731 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:12.732 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:12.808 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:47:12.815 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:13.056 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:13.247 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:13.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:47:13.372 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:13.390 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 145 k 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 432 k 2024-09-13T13:47:13.391 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:13.506 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:47:13.507 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-09-13T13:47:13.508 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:13.509 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 19 M 2024-09-13T13:47:13.509 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 159 M 2024-09-13T13:47:13.509 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:13.559 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:13.600 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 697 kB/s | 145 kB 00:00 2024-09-13T13:47:13.600 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:13.601 INFO:teuthology.orchestra.run.smithi027.stdout:Total 691 kB/s | 145 kB 00:00 2024-09-13T13:47:13.601 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:13.609 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:13.609 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:13.669 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:13.669 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:13.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-09-13T13:47:13.817 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:13.838 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:13.909 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:13.935 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:13.935 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:13.935 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:47:13.936 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:13.972 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:47:13.973 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:13.974 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 145 k 2024-09-13T13:47:13.974 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 432 k 2024-09-13T13:47:13.974 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:14.178 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 712 kB/s | 145 kB 00:00 2024-09-13T13:47:14.179 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:14.179 INFO:teuthology.orchestra.run.smithi122.stdout:Total 705 kB/s | 145 kB 00:00 2024-09-13T13:47:14.179 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:14.187 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:14.187 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:14.243 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:14.243 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:14.323 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 228 kB/s | 34 kB 00:00 2024-09-13T13:47:14.348 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 94 kB/s | 16 kB 00:00 2024-09-13T13:47:14.378 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:14.380 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:14.462 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:14.472 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:14.487 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:14.488 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:14.488 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:47:14.488 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:14.490 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-13T13:47:14.513 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:14.514 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:47:14.515 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:14.540 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-09-13T13:47:14.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:47:14.615 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-09-13T13:47:14.641 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-09-13T13:47:14.741 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-09-13T13:47:14.766 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-13T13:47:14.791 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:14.834 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:14.934 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 19 MB/s | 14 MB 00:00 2024-09-13T13:47:14.967 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.8 MB/s | 863 kB 00:00 2024-09-13T13:47:14.992 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2024-09-13T13:47:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:14.996 INFO:teuthology.orchestra.run.smithi159.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-13T13:47:15.049 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:15.069 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:15.069 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:15.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-09-13T13:47:15.127 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:15.259 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:47:15.260 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:15.261 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-13T13:47:15.262 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:15.262 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 19 M 2024-09-13T13:47:15.262 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 159 M 2024-09-13T13:47:15.262 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:15.338 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:15.338 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:15.399 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:47:15.400 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:15.400 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:15.400 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:15.400 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:15.400 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:15.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-09-13T13:47:15.668 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:15.762 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:15.801 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 218 kB/s | 34 kB 00:00 2024-09-13T13:47:15.806 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:15.807 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:47:15.807 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:15.826 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 90 kB/s | 16 kB 00:00 2024-09-13T13:47:15.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:47:15.929 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:15.985 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-13T13:47:16.009 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:47:16.045 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:47:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:47:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:47:16.152 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-13T13:47:16.206 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:16.225 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:47:16.300 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:16.340 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:16.341 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:16.341 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:16.349 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:47:16.352 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 20 MB/s | 14 MB 00:00 2024-09-13T13:47:16.386 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-09-13T13:47:16.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:47:16.428 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 288 kB/s | 19 kB 00:00 2024-09-13T13:47:16.441 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:47:16.460 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:03 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:16.461 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-13T13:47:16.487 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T13:47:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:47:16.520 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-13T13:47:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:47:16.579 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-13T13:47:16.600 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:16.601 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:16.601 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:47:16.602 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-09-13T13:47:16.603 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 19 M 2024-09-13T13:47:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 159 M 2024-09-13T13:47:16.604 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:16.982 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:17.071 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 795 kB/s | 863 kB 00:01 2024-09-13T13:47:17.075 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:17.076 INFO:teuthology.orchestra.run.smithi100.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-13T13:47:17.114 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:47:17.115 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-13T13:47:17.116 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:17.117 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 19 M 2024-09-13T13:47:17.117 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 159 M 2024-09-13T13:47:17.117 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:17.125 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:17.144 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:17.144 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:17.295 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 241 kB/s | 34 kB 00:00 2024-09-13T13:47:17.312 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 104 kB/s | 16 kB 00:00 2024-09-13T13:47:17.403 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:17.403 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:17.454 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-13T13:47:17.496 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-09-13T13:47:17.547 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-13T13:47:17.571 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-13T13:47:17.614 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-09-13T13:47:17.672 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-13T13:47:17.697 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T13:47:17.748 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-13T13:47:17.773 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-13T13:47:17.923 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 18 MB/s | 14 MB 00:00 2024-09-13T13:47:17.928 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:17.928 INFO:teuthology.orchestra.run.smithi027.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-13T13:47:17.978 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:17.998 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 246 kB/s | 34 kB 00:00 2024-09-13T13:47:17.999 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:17.999 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:18.024 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 100 kB/s | 16 kB 00:00 2024-09-13T13:47:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:47:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:47:18.161 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:47:18.202 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:47:18.224 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-13T13:47:18.270 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:18.270 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:18.323 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:47:18.382 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-13T13:47:18.450 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:47:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:47:18.558 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 20 MB/s | 14 MB 00:00 2024-09-13T13:47:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:47:18.600 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-09-13T13:47:18.625 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 301 kB/s | 19 kB 00:00 2024-09-13T13:47:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:18.657 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:47:18.658 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-09-13T13:47:18.692 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-13T13:47:18.726 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 460 kB/s | 31 kB 00:00 2024-09-13T13:47:18.751 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-13T13:47:18.801 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-13T13:47:18.806 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:18.806 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-13T13:47:18.857 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:18.872 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:18.877 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:18.877 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:18.958 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:47:18.993 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:47:19.041 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:47:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:47:19.161 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:19.162 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:19.192 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:47:19.328 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:47:19.418 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:47:19.467 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:47:19.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:19.602 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:47:19.754 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:19.765 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:47:19.833 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:19.835 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:47:19.870 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:47:19.917 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:47:19.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:47:20.052 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:47:20.182 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:47:20.276 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:47:20.335 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:47:20.364 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:20.405 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:47:21.771 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:21.771 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:47:21.772 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:47:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:47:21.921 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:22.084 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:47:22.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:47:22.086 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:47:22.086 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:22.086 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:22.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:47:22.905 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:08 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:23.018 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:47:23.039 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:23.040 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:47:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================= 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-09-13T13:47:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:23.043 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 38 M 2024-09-13T13:47:23.043 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 212 M 2024-09-13T13:47:23.043 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:23.073 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:23.549 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 917 kB/s | 191 kB 00:00 2024-09-13T13:47:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:47:23.607 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-13T13:47:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:23.708 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 33 kB 00:00 2024-09-13T13:47:23.799 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-13T13:47:23.816 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-09-13T13:47:23.925 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-13T13:47:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:47:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:47:24.034 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:47:24.035 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-09-13T13:47:24.084 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-09-13T13:47:24.151 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-13T13:47:24.176 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.7 MB/s | 250 kB 00:00 2024-09-13T13:47:24.301 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-09-13T13:47:24.365 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:47:24.366 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:47:24.367 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:47:24.367 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:47:24.367 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:24.367 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:24.552 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-13T13:47:24.553 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:24.553 INFO:teuthology.orchestra.run.smithi159.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-13T13:47:24.622 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:47:24.895 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:24.895 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:47:24.896 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:47:24.909 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:24.935 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:24.935 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:25.185 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:10 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:25.220 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:47:25.221 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:47:25.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:25.222 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:25.316 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:25.317 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T13:47:25.317 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:25.317 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T13:47:25.317 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:47:25.318 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:25.319 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 38 M 2024-09-13T13:47:25.320 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 212 M 2024-09-13T13:47:25.320 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:25.374 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:25.375 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:25.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:47:25.562 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:47:25.563 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:47:25.786 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 761 kB/s | 191 kB 00:00 2024-09-13T13:47:25.878 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-13T13:47:25.891 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:47:25.891 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:25.891 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:25.891 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:47:25.891 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:25.892 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 801 kB 00:00 2024-09-13T13:47:26.045 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-46 15 MB/s | 7.4 MB 00:00 2024-09-13T13:47:26.064 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:13 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:26.120 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 104 kB/s | 18 kB 00:00 2024-09-13T13:47:26.146 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 385 kB/s | 38 kB 00:00 2024-09-13T13:47:26.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:47:26.161 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:26.203 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:26.204 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-13T13:47:26.204 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:26.204 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:47:26.205 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-09-13T13:47:26.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:26.207 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 38 M 2024-09-13T13:47:26.207 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 212 M 2024-09-13T13:47:26.207 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:26.229 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-13T13:47:26.241 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:47:26.283 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:26.385 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:26.418 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:47:26.454 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-09-13T13:47:26.546 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-13T13:47:26.601 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:47:26.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:47:26.693 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:47:26.707 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:11 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:26.741 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 951 kB/s | 191 kB 00:00 2024-09-13T13:47:26.821 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-09-13T13:47:26.839 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:26.840 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:47:26.841 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-13T13:47:26.842 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:26.843 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 38 M 2024-09-13T13:47:26.843 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 212 M 2024-09-13T13:47:26.843 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:26.847 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-13T13:47:26.914 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-09-13T13:47:27.014 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-13T13:47:27.031 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-13T13:47:27.115 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-09-13T13:47:27.165 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-13T13:47:27.240 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-09-13T13:47:27.265 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-13T13:47:27.322 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-13T13:47:27.374 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 14 MB/s | 4.9 MB 00:00 2024-09-13T13:47:27.386 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:47:27.475 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-13T13:47:27.489 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-13T13:47:27.497 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:47:27.574 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-13T13:47:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-13T13:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:27.575 INFO:teuthology.orchestra.run.smithi027.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-13T13:47:27.642 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 302 kB/s | 191 kB 00:00 2024-09-13T13:47:27.742 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-13T13:47:27.767 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 145 kB/s | 18 kB 00:00 2024-09-13T13:47:27.834 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-13T13:47:27.860 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 327 kB/s | 38 kB 00:00 2024-09-13T13:47:27.885 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.7 MB/s | 245 kB 00:00 2024-09-13T13:47:27.936 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-09-13T13:47:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:27.967 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:27.967 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:28.102 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-13T13:47:28.194 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 12 MB/s | 4.9 MB 00:00 2024-09-13T13:47:28.345 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-09-13T13:47:28.345 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:28.345 INFO:teuthology.orchestra.run.smithi122.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-13T13:47:28.426 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:28.427 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:28.702 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:28.728 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:28.728 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:29.167 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:29.167 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:29.314 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-13T13:47:29.314 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:29.315 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.6 MB/s | 38 MB 00:03 2024-09-13T13:47:29.333 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:29.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:47:29.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:29.545 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:47:29.670 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:29.696 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:29.696 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:29.760 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:47:29.774 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:47:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:29.806 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:47:29.893 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:47:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:47:30.076 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:30.126 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:30.126 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:30.175 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:30.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:47:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:47:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:47:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:47:30.596 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:47:30.708 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:47:30.885 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:30.968 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:47:31.012 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:31.112 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:31.155 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:47:31.168 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:47:31.288 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:47:31.339 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:47:31.370 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:47:31.452 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:47:32.028 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:32.116 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:47:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:47:32.362 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:32.363 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:47:32.504 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:47:32.761 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:32.762 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:33.119 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:47:33.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:47:33.155 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:33.459 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:33.527 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:47:33.559 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:33.712 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:19 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:33.850 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:47:33.852 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:33.853 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2024-09-13T13:47:33.854 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:33.854 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.2 M 2024-09-13T13:47:33.854 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-09-13T13:47:33.854 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:34.299 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 293 kB/s | 26 kB 00:00 2024-09-13T13:47:34.324 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 170 kB/s | 19 kB 00:00 2024-09-13T13:47:34.383 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 287 kB/s | 49 kB 00:00 2024-09-13T13:47:34.433 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-13T13:47:34.458 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T13:47:34.492 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-09-13T13:47:34.493 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:47:34.520 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:34.525 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-13T13:47:34.551 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-13T13:47:34.584 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-13T13:47:34.601 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.4 MB/s | 59 kB 00:00 2024-09-13T13:47:34.627 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-13T13:47:34.652 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-13T13:47:34.685 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-13T13:47:34.687 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:34.687 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-13T13:47:34.751 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:34.766 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:34.766 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:34.854 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:34.856 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:35.537 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:47:35.612 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:47:35.676 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:47:35.681 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:35.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:47:35.681 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:35.682 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:47:35.713 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:47:35.754 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:47:35.866 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:47:35.976 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:47:36.031 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.129 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:36.130 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:47:36.203 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:36.204 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:47:36.525 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:47:36.624 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:36.625 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:36.774 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:47:36.828 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:47:36.862 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:47:36.910 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:47:36.940 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:37.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:47:37.121 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:47:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:47:37.262 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:37.262 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:37.262 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:37.262 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:47:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:37.264 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.2 M 2024-09-13T13:47:37.265 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-09-13T13:47:37.265 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:37.604 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:37.699 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:47:37.699 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:47:37.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:37.701 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:47:37.701 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:47:37.701 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:37.701 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:47:37.746 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:47:37.747 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:47:37.748 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-09-13T13:47:37.749 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:37.750 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.2 M 2024-09-13T13:47:37.750 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-09-13T13:47:37.750 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:37.963 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 173 kB/s | 19 kB 00:00 2024-09-13T13:47:37.980 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 26 kB 00:00 2024-09-13T13:47:38.005 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 321 kB/s | 49 kB 00:00 2024-09-13T13:47:38.072 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T13:47:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:47:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:38.083 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:38.083 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:47:38.084 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:47:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:47:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:47:38.085 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:38.085 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:38.089 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T13:47:38.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:47:38.115 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.2 MB/s | 54 kB 00:00 2024-09-13T13:47:38.140 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-09-13T13:47:38.165 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 292 kB/s | 14 kB 00:00 2024-09-13T13:47:38.190 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.3 MB/s | 220 kB 00:00 2024-09-13T13:47:38.216 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T13:47:38.241 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T13:47:38.266 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-13T13:47:38.277 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 19 kB 00:00 2024-09-13T13:47:38.302 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 26 kB 00:00 2024-09-13T13:47:38.308 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-13T13:47:38.310 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:38.310 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-13T13:47:38.318 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:47:38.327 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 273 kB/s | 49 kB 00:00 2024-09-13T13:47:38.381 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:38.397 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:38.397 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:38.461 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-13T13:47:38.486 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-09-13T13:47:38.495 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:38.497 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:38.520 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-13T13:47:38.553 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-13T13:47:38.578 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-13T13:47:38.604 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-13T13:47:38.638 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-13T13:47:38.663 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-13T13:47:38.663 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:23 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:38.688 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-13T13:47:38.763 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-09-13T13:47:38.766 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:38.767 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-13T13:47:38.799 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:47:38.801 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-09-13T13:47:38.802 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:38.803 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.2 M 2024-09-13T13:47:38.803 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-09-13T13:47:38.803 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:38.830 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:38.846 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:38.846 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:38.890 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:38.934 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:38.934 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:39.027 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:39.031 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:47:39.032 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-09-13T13:47:39.033 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:39.034 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 7.0 M 2024-09-13T13:47:39.034 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 33 M 2024-09-13T13:47:39.034 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:39.054 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 287 kB/s | 49 kB 00:00 2024-09-13T13:47:39.171 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 89 kB/s | 26 kB 00:00 2024-09-13T13:47:39.196 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 61 kB/s | 19 kB 00:00 2024-09-13T13:47:39.198 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:47:39.254 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:47:39.309 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:47:39.344 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:47:39.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:47:39.396 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 732 kB/s | 154 kB 00:00 2024-09-13T13:47:39.441 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:39.480 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-13T13:47:39.494 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:47:39.496 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-13T13:47:39.522 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-13T13:47:39.572 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-13T13:47:39.603 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:47:39.617 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:47:39.655 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:47:39.655 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 608 kB/s | 279 kB 00:00 2024-09-13T13:47:39.672 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:47:39.689 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 350 kB/s | 222 kB 00:00 2024-09-13T13:47:39.723 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 356 kB/s | 54 kB 00:00 2024-09-13T13:47:39.736 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:47:39.756 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 319 kB/s | 32 kB 00:00 2024-09-13T13:47:39.774 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:47:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 142 kB/s | 14 kB 00:00 2024-09-13T13:47:39.815 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:47:39.822 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 390 kB/s | 248 kB 00:00 2024-09-13T13:47:39.823 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-13T13:47:39.857 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-13T13:47:39.891 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-09-13T13:47:39.916 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-google-auth-2.34.0-1.el9.noarc 1.2 MB/s | 220 kB 00:00 2024-09-13T13:47:39.916 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:47:39.920 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:39.920 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-09-13T13:47:39.986 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:40.001 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:40.001 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:40.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:47:40.066 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:47:40.090 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:40.090 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:40.416 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:47:40.468 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:47:40.502 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:47:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:47:40.579 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:40.601 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:47:40.788 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:47:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:47:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:47:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:47:40.904 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:47:40.926 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:47:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:47:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:47:41.121 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:47:41.230 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:47:41.279 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:47:41.356 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:41.356 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:47:41.356 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:47:41.356 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:47:41.357 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:47:41.702 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:47:41.702 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:41.702 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:41.702 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:47:41.703 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:47:41.704 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:41.704 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:41.787 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:41.787 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:47:41.787 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:47:41.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:47:41.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:47:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:47:42.042 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:47:42.072 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.1 MB/s | 6.0 MB 00:02 2024-09-13T13:47:42.074 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:42.074 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.3 MB/s | 7.0 MB 00:03 2024-09-13T13:47:42.075 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:47:42.113 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:42.114 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:42.123 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:47:42.145 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:42.147 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:42.159 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:42.159 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:42.260 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:42.260 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:42.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:47:42.519 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:42.562 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:42.659 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:42.659 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:42.659 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:47:42.660 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:42.661 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 7.0 M 2024-09-13T13:47:42.662 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 33 M 2024-09-13T13:47:42.662 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:42.737 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:47:42.790 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:47:42.914 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:47:42.924 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:47:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:47:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:47:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:47:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:47:42.925 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:47:43.046 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:47:43.047 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-09-13T13:47:43.048 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:43.049 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2024-09-13T13:47:43.049 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-09-13T13:47:43.049 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:43.254 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:47:43.254 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:43.254 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:43.254 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:43.254 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:43.255 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:43.307 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:47:43.319 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 734 kB/s | 154 kB 00:00 2024-09-13T13:47:43.349 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 687 kB/s | 154 kB 00:00 2024-09-13T13:47:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 995 kB/s | 248 kB 00:00 2024-09-13T13:47:43.393 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:47:43.428 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-13T13:47:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 691 kB/s | 58 kB 00:00 2024-09-13T13:47:43.460 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:47:43.461 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-13T13:47:43.483 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-13T13:47:43.486 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:43.495 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-13T13:47:43.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:47:43.508 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-13T13:47:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-13T13:47:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-13T13:47:43.673 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:43.684 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:43.684 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:43.784 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:43.785 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:43.995 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 280 kB/s | 248 kB 00:00 2024-09-13T13:47:44.070 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:44.086 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:44.203 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:44.203 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:44.204 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 7.0 M 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-09-13T13:47:44.205 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:44.248 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:47:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:47:44.302 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:47:44.370 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-09-13T13:47:44.373 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:44.373 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-13T13:47:44.435 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-13T13:47:44.449 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:44.462 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:44.462 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:44.527 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 689 kB/s | 154 kB 00:00 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:44.541 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:44.569 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:44.569 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:44.577 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T13:47:44.610 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.1 MB/s | 548 kB 00:00 2024-09-13T13:47:44.644 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-13T13:47:44.769 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2024-09-13T13:47:44.819 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:47:44.893 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:44.904 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:47:44.981 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:47:45.015 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:45.064 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:47:45.124 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:47:45.343 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:45.477 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:45.477 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-09-13T13:47:45.478 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:45.479 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 979 k 2024-09-13T13:47:45.479 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.9 M 2024-09-13T13:47:45.479 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:45.632 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 742 kB/s | 80 kB 00:00 2024-09-13T13:47:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:47:45.771 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:47:45.784 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-13T13:47:45.784 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:45.784 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.1 MB/s | 979 kB 00:00 2024-09-13T13:47:45.795 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:47:45.815 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:47:45.816 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:45.816 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:45.847 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:47:45.861 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-09-13T13:47:45.864 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:45.864 INFO:teuthology.orchestra.run.smithi100.stdout:Total 4.2 MB/s | 7.0 MB 00:01 2024-09-13T13:47:45.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:45.898 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:45.899 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:45.937 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:45.949 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:45.949 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:46.049 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:46.051 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:46.079 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:47:46.079 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:46.079 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:46.079 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:46.079 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:46.080 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:46.083 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:46.210 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:47:46.281 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:46.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-09-13T13:47:46.364 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:46.534 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:47:46.579 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:47:46.632 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:47:46.884 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:47:46.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:47:46.924 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:47:46.924 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:47:46.924 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:46.924 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:46.992 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:46.993 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:47:47.019 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:47.020 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:47.021 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-09-13T13:47:47.021 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:47.021 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 979 k 2024-09-13T13:47:47.021 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.9 M 2024-09-13T13:47:47.021 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:47:47.145 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 804 kB/s | 80 kB 00:00 2024-09-13T13:47:47.190 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:47:47.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-09-13T13:47:47.258 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:47:47.283 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:47.290 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:47:47.290 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:47.290 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:47.291 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:47:47.291 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:47.291 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:47.320 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T13:47:47.320 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:47.321 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.2 MB/s | 979 kB 00:00 2024-09-13T13:47:47.332 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:47.353 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:47.435 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:47.435 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:47.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-volume 2024-09-13T13:47:47.638 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:47:47.814 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:47.843 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:47.954 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:47.955 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-09-13T13:47:47.956 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:47.956 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 979 k 2024-09-13T13:47:47.956 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.9 M 2024-09-13T13:47:47.956 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:48.076 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:47:48.076 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:47:48.076 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:47:48.077 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:47:48.077 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:47:48.077 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:47:48.159 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:48.239 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 543 kB/s | 80 kB 00:00 2024-09-13T13:47:48.295 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:47:48.296 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 285 k 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.3 M 2024-09-13T13:47:48.297 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:47:48.343 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:47:48.344 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:48.344 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:48.364 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T13:47:48.365 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:48.365 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-13T13:47:48.377 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:48.401 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:48.401 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:48.489 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:48.490 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:48.521 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:48.521 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:48.521 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:48.522 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:48.522 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:48.522 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:47:48.530 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:48.530 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:48.556 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:48.557 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:48.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-09-13T13:47:48.684 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:48.773 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:48.806 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:47:48.806 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:48.806 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:48.806 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:47:48.806 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:48.807 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:48.821 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:47:48.839 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:48.856 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:48.856 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:48.856 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:47:48.856 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:48.900 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:49.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-09-13T13:47:49.170 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:49.302 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:49.302 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-13T13:47:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:49.304 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 979 k 2024-09-13T13:47:49.304 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.9 M 2024-09-13T13:47:49.304 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:49.479 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 541 kB/s | 80 kB 00:00 2024-09-13T13:47:49.596 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-13T13:47:49.597 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:49.597 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.3 MB/s | 979 kB 00:00 2024-09-13T13:47:49.608 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:49.622 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:49.622 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:47:49.630 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:49.630 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:49.673 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:49.710 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:49.711 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:49.809 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:49.810 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 285 k 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.3 M 2024-09-13T13:47:49.811 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:49.890 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:49.907 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:49.922 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:47:49.922 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:49.922 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:47:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:49.923 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:50.024 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-13T13:47:50.025 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:50.025 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-13T13:47:50.025 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:50.032 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:47:50.033 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:50.033 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:50.066 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:50.067 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:50.103 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:50.168 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2024-09-13T13:47:50.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-volume 2024-09-13T13:47:50.288 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:50.385 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:50.401 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:50.402 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:50.402 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:47:50.402 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:50.752 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:50.764 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:47:50.765 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:47:50.805 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:50.886 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:50.886 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 127 k 2024-09-13T13:47:50.887 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 456 k 2024-09-13T13:47:50.888 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:50.944 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:50.945 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-13T13:47:50.945 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:50.945 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 285 k 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 1.3 M 2024-09-13T13:47:50.946 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:51.054 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:51.085 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 643 kB/s | 127 kB 00:00 2024-09-13T13:47:51.085 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:51.085 INFO:teuthology.orchestra.run.smithi159.stdout:Total 638 kB/s | 127 kB 00:00 2024-09-13T13:47:51.086 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:51.091 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:51.091 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:51.154 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:51.155 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:51.177 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:51.178 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:51.178 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:51.178 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:51.224 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:51.225 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:51.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-09-13T13:47:51.408 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:51.428 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:51.429 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:51.429 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:51.429 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:51.429 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:51.429 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:51.482 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:51.596 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:51.613 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:51.613 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:51.613 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:47:51.613 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:51.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-09-13T13:47:51.827 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:51.946 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:52.081 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:52.082 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:52.082 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:52.082 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:52.082 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:52.082 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:52.083 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:52.084 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 285 k 2024-09-13T13:47:52.085 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.3 M 2024-09-13T13:47:52.085 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:52.318 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:52.318 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:52.318 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T13:47:52.319 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:52.321 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:52.327 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:52.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2024-09-13T13:47:52.357 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:52.357 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:52.458 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:52.459 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:52.459 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T13:47:52.459 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:52.459 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:52.459 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 456 k 2024-09-13T13:47:52.460 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:52.571 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:52.583 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:52.636 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:52.653 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:52.653 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:47:52.653 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:47:52.653 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:52.680 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 576 kB/s | 127 kB 00:00 2024-09-13T13:47:52.681 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:52.681 INFO:teuthology.orchestra.run.smithi122.stdout:Total 571 kB/s | 127 kB 00:00 2024-09-13T13:47:52.681 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:52.687 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:52.687 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:52.750 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:52.751 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:52.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-09-13T13:47:52.906 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:52.914 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:53.006 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:53.050 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:53.051 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:53.052 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:53.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2024-09-13T13:47:53.438 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:53.495 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:53.636 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:53.637 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:47:53.638 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:53.638 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 127 k 2024-09-13T13:47:53.638 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 456 k 2024-09-13T13:47:53.638 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:53.682 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:53.704 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:47:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:53.762 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:47:53.762 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:53.816 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 32 k 2024-09-13T13:47:53.817 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 142 k 2024-09-13T13:47:53.818 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:47:53.849 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 601 kB/s | 127 kB 00:00 2024-09-13T13:47:53.849 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:53.849 INFO:teuthology.orchestra.run.smithi027.stdout:Total 596 kB/s | 127 kB 00:00 2024-09-13T13:47:53.850 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:53.856 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:53.856 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:53.924 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:53.924 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:53.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-09-13T13:47:53.963 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-13T13:47:53.964 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:53.964 INFO:teuthology.orchestra.run.smithi159.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-13T13:47:53.964 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:47:53.968 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:47:53.968 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:47:53.984 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:47:53.985 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:47:54.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-09-13T13:47:54.084 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:54.095 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:47:54.192 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.203 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.531 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:54.596 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.622 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:54.631 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.640 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:54.666 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:54.667 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:54.668 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:54.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-09-13T13:47:54.774 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:47:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:54.776 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 127 k 2024-09-13T13:47:54.776 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-09-13T13:47:54.776 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:47:54.821 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:54.890 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:54.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:54.891 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:54.891 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:54.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:54.891 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:54.982 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 616 kB/s | 127 kB 00:00 2024-09-13T13:47:54.983 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:54.983 INFO:teuthology.orchestra.run.smithi100.stdout:Total 611 kB/s | 127 kB 00:00 2024-09-13T13:47:54.983 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:54.988 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:54.988 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:55.011 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2024-09-13T13:47:55.053 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:55.054 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:55.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-09-13T13:47:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:55.304 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:55.321 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:55.436 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:47:55.437 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:47:55.438 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:47:55.438 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:55.438 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 32 k 2024-09-13T13:47:55.438 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 142 k 2024-09-13T13:47:55.438 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:47:55.583 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:55.583 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-13T13:47:55.584 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:55.584 INFO:teuthology.orchestra.run.smithi122.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-13T13:47:55.585 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:47:55.588 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:47:55.588 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:47:55.606 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:47:55.606 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:47:55.674 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:47:55.718 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:55.719 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:55.719 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:55.750 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:55.780 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2024-09-13T13:47:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:55.807 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:55.844 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:55.890 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:55.891 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:47:55.891 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:55.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-09-13T13:47:56.069 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:56.069 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:56.069 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:56.070 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:56.070 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:56.070 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:56.188 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:56.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-09-13T13:47:56.347 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:56.411 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:56.411 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:56.411 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:47:56.411 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:56.411 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:47:56.412 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:56.437 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:56.482 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:56.483 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:56.483 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:56.548 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:56.550 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2024-09-13T13:47:56.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-09-13T13:47:56.687 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:47:56.688 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 32 k 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 142 k 2024-09-13T13:47:56.689 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:47:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 203 kB/s | 32 kB 00:00 2024-09-13T13:47:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-13T13:47:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:47:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:47:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:47:56.870 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:47:56.870 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:47:56.882 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:56.951 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:47:56.973 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:57.017 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:57.018 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:47:57.018 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:57.070 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:57.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-09-13T13:47:57.119 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:57.165 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:57.211 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:57.254 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:57.255 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:57.256 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:57.256 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:57.297 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:57.297 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:57.298 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:57.317 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2024-09-13T13:47:57.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-09-13T13:47:57.421 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:57.649 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:47:57.657 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:57.783 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:47:57.784 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 32 k 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 142 k 2024-09-13T13:47:57.785 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:47:57.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-09-13T13:47:57.887 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:57.922 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:57.944 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 199 kB/s | 32 kB 00:00 2024-09-13T13:47:57.945 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:47:57.945 INFO:teuthology.orchestra.run.smithi100.stdout:Total 197 kB/s | 32 kB 00:00 2024-09-13T13:47:57.945 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:47:57.948 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:47:57.949 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:47:57.967 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:47:57.967 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:47:57.977 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:58.011 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:58.021 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:58.022 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:58.022 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:58.054 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:58.055 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:58.055 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:58.060 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:47:58.083 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2024-09-13T13:47:58.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-09-13T13:47:58.158 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:58.452 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:58.545 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:58.591 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:58.592 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:47:58.592 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:58.643 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:58.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-09-13T13:47:58.695 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:58.732 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:58.775 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:58.776 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:58.776 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:58.785 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:47:58.800 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:58.828 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:58.829 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:58.829 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:58.839 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2024-09-13T13:47:58.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-09-13T13:47:59.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-09-13T13:47:59.244 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:47:59.338 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:59.385 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:47:59.386 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:47:59.386 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:47:59.408 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:47:59.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-09-13T13:47:59.457 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:59.497 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:59.541 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:47:59.542 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-09-13T13:47:59.542 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:47:59.546 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:59.570 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:47:59.589 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:47:59.590 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:47:59.590 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:47:59.607 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2024-09-13T13:47:59.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-09-13T13:47:59.659 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:47:59.703 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:47:59.704 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:47:59.704 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:47:59.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-09-13T13:48:00.041 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:00.133 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:00.175 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:48:00.178 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:00.180 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:48:00.180 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:00.226 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:00.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-09-13T13:48:00.309 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:48:00.309 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:00.309 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 85 k 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 231 k 2024-09-13T13:48:00.310 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:48:00.316 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:00.333 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:00.359 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:48:00.361 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:48:00.361 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:48:00.423 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:00.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-09-13T13:48:00.467 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:00.468 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:48:00.468 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:00.483 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 494 kB/s | 85 kB 00:00 2024-09-13T13:48:00.483 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:00.483 INFO:teuthology.orchestra.run.smithi159.stdout:Total 491 kB/s | 85 kB 00:00 2024-09-13T13:48:00.484 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:48:00.490 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:48:00.490 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:48:00.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-09-13T13:48:00.554 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:48:00.554 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:48:00.697 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:48:00.781 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:00.826 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:00.918 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:00.964 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:00.965 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:48:00.965 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:00.989 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:01.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-09-13T13:48:01.079 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:01.106 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:01.122 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:48:01.123 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-13T13:48:01.123 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:48:01.133 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:01.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-09-13T13:48:01.195 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:01.238 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:01.239 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:48:01.239 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:01.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:01.389 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:48:01.619 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:01.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2024-09-13T13:48:01.714 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:01.754 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:01.760 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:01.760 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:48:01.761 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:01.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-09-13T13:48:01.862 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:01.887 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:01.888 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:48:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:01.889 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 85 k 2024-09-13T13:48:01.889 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 231 k 2024-09-13T13:48:01.889 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:48:01.951 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:01.994 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:01.995 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:48:01.995 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:02.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-09-13T13:48:02.105 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 395 kB/s | 85 kB 00:00 2024-09-13T13:48:02.106 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:02.107 INFO:teuthology.orchestra.run.smithi122.stdout:Total 392 kB/s | 85 kB 00:00 2024-09-13T13:48:02.107 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:48:02.113 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:48:02.113 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:48:02.177 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:48:02.177 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:48:02.253 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:48:02.331 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:48:02.395 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:48:02.395 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 3.2 M 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 14 M 2024-09-13T13:48:02.396 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:48:02.415 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:02.508 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:02.554 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:02.555 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-13T13:48:02.555 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:02.621 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:02.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-09-13T13:48:02.711 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:02.720 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-13T13:48:02.721 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:02.721 INFO:teuthology.orchestra.run.smithi159.stdout:Total 9.7 MB/s | 3.2 MB 00:00 2024-09-13T13:48:02.721 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:48:02.730 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:48:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:48:02.754 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:02.755 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:48:02.755 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:02.803 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:02.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-09-13T13:48:02.818 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:48:02.818 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:48:03.085 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:03.086 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:03.086 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:48:03.086 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:03.086 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:03.086 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:48:03.219 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:03.278 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:48:03.346 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:03.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-09-13T13:48:03.361 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:03.362 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:03.362 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 85 k 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 231 k 2024-09-13T13:48:03.363 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:48:03.371 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:03.371 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:48:03.372 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:48:03.372 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:03.372 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:03.372 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:03.378 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:03.468 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:48:03.511 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:03.512 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-13T13:48:03.513 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:03.559 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 435 kB/s | 85 kB 00:00 2024-09-13T13:48:03.559 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:03.560 INFO:teuthology.orchestra.run.smithi027.stdout:Total 432 kB/s | 85 kB 00:00 2024-09-13T13:48:03.560 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:48:03.567 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:48:03.567 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:48:03.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-09-13T13:48:03.634 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:48:03.634 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:48:03.784 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:48:03.891 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:03.928 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:04.063 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:48:04.064 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:04.065 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:48:04.065 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:04.065 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.2 M 2024-09-13T13:48:04.065 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 14 M 2024-09-13T13:48:04.065 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:48:04.141 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:04.273 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:48:04.274 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 85 k 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 231 k 2024-09-13T13:48:04.275 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:48:04.277 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:04.391 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.7 MB/s | 3.2 MB 00:00 2024-09-13T13:48:04.391 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:04.392 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.7 MB/s | 3.2 MB 00:00 2024-09-13T13:48:04.392 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:48:04.401 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:48:04.401 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:48:04.420 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:04.420 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:04.421 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:48:04.421 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:04.421 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:04.421 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:48:04.472 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 432 kB/s | 85 kB 00:00 2024-09-13T13:48:04.473 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:04.473 INFO:teuthology.orchestra.run.smithi100.stdout:Total 429 kB/s | 85 kB 00:00 2024-09-13T13:48:04.473 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:48:04.480 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:48:04.480 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:48:04.487 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:48:04.487 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:48:04.543 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:48:04.544 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:48:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:04.553 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:48:04.553 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:04.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:04.554 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:04.672 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2024-09-13T13:48:04.689 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:48:04.795 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:04.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-09-13T13:48:04.947 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:48:05.016 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:05.042 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:05.232 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:05.258 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 01:46:14 PM UTC. 2024-09-13T13:48:05.393 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-09-13T13:48:05.393 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:05.393 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:05.393 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 170 k 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 490 k 2024-09-13T13:48:05.394 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-09-13T13:48:05.404 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:05.478 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:05.546 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:05.546 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.2 M 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 14 M 2024-09-13T13:48:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:48:05.598 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 836 kB/s | 170 kB 00:00 2024-09-13T13:48:05.599 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:05.599 INFO:teuthology.orchestra.run.smithi159.stdout:Total 830 kB/s | 170 kB 00:00 2024-09-13T13:48:05.600 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-09-13T13:48:05.608 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-09-13T13:48:05.608 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-09-13T13:48:05.671 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-09-13T13:48:05.671 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-09-13T13:48:05.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-09-13T13:48:05.830 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-09-13T13:48:05.876 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-13T13:48:05.877 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:05.877 INFO:teuthology.orchestra.run.smithi027.stdout:Total 9.6 MB/s | 3.2 MB 00:00 2024-09-13T13:48:05.877 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:48:05.887 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:48:05.887 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:48:05.929 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:05.979 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:48:05.979 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:48:06.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:06.175 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:06.175 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:48:06.175 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:06.176 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:06.176 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:48:06.340 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:06.388 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:06.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-09-13T13:48:06.462 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:48:06.531 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:48:06.532 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:06.533 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:48:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:06.533 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.2 M 2024-09-13T13:48:06.533 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 14 M 2024-09-13T13:48:06.533 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:48:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:06.623 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-09-13T13:48:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:06.623 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:06.624 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-09-13T13:48:06.630 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:06.657 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:06.657 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:48:06.658 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:48:06.658 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:06.658 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:06.658 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:06.852 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-13T13:48:06.852 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:06.853 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.9 MB/s | 3.2 MB 00:00 2024-09-13T13:48:06.853 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:48:06.859 DEBUG:teuthology.parallel:result is None 2024-09-13T13:48:06.861 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:48:06.862 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:48:06.948 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:48:06.949 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:48:07.009 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:07.146 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-13T13:48:07.146 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 170 k 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 490 k 2024-09-13T13:48:07.147 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-13T13:48:07.386 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 713 kB/s | 170 kB 00:00 2024-09-13T13:48:07.387 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:07.387 INFO:teuthology.orchestra.run.smithi122.stdout:Total 709 kB/s | 170 kB 00:00 2024-09-13T13:48:07.387 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-13T13:48:07.395 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-13T13:48:07.395 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-13T13:48:07.396 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:48:07.444 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:07.462 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-13T13:48:07.463 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:48:07.471 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:07.595 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:07.596 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:48:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:07.596 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:07.596 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-13T13:48:07.736 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:07.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-09-13T13:48:08.216 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:08.464 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 01:46:13 PM UTC. 2024-09-13T13:48:08.471 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:08.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:08.472 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-13T13:48:08.472 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:08.472 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:08.472 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-13T13:48:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:08.543 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:48:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:08.543 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:08.602 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-13T13:48:08.604 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:08.604 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:08.604 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:08.604 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-13T13:48:08.604 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:48:08.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:08.605 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-13T13:48:08.605 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-13T13:48:08.605 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-13T13:48:08.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:08.606 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 170 k 2024-09-13T13:48:08.606 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 490 k 2024-09-13T13:48:08.606 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-13T13:48:08.691 DEBUG:teuthology.parallel:result is None 2024-09-13T13:48:08.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-09-13T13:48:08.817 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 801 kB/s | 170 kB 00:00 2024-09-13T13:48:08.818 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:08.818 INFO:teuthology.orchestra.run.smithi027.stdout:Total 796 kB/s | 170 kB 00:00 2024-09-13T13:48:08.818 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-13T13:48:08.826 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-13T13:48:08.826 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-13T13:48:08.894 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-13T13:48:08.894 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-13T13:48:09.052 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-13T13:48:09.139 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:09.353 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 01:46:15 PM UTC. 2024-09-13T13:48:09.491 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-13T13:48:09.492 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:09.492 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-13T13:48:09.492 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 170 k 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 490 k 2024-09-13T13:48:09.493 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-13T13:48:09.539 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:09.731 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 718 kB/s | 170 kB 00:00 2024-09-13T13:48:09.731 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:09.731 INFO:teuthology.orchestra.run.smithi100.stdout:Total 713 kB/s | 170 kB 00:00 2024-09-13T13:48:09.732 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-13T13:48:09.739 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-13T13:48:09.739 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-13T13:48:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-13T13:48:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-13T13:48:09.803 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-13T13:48:09.803 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-13T13:48:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-13T13:48:10.059 DEBUG:teuthology.parallel:result is None 2024-09-13T13:48:10.063 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:10.494 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:10.740 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:48:10.740 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:10.740 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-13T13:48:10.740 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:48:10.740 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:10.741 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-13T13:48:11.000 DEBUG:teuthology.parallel:result is None 2024-09-13T13:48:11.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:11.155 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:48:11.178 INFO:teuthology.orchestra.run.smithi027.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.178 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.179 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:48:11.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:11.326 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:48:11.348 INFO:teuthology.orchestra.run.smithi100.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.348 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.348 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:48:11.350 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:11.493 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:48:11.518 INFO:teuthology.orchestra.run.smithi122.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.518 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.518 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:48:11.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:11.668 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:48:11.695 INFO:teuthology.orchestra.run.smithi159.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.695 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:48:11.695 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:48:11.697 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T13:48:11.697 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:11.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:48:11.725 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:11.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:48:11.756 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:11.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:48:11.789 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:11.789 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:48:11.821 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T13:48:11.821 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:11.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:48:11.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:48:11.912 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:11.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:48:11.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:48:12.006 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:12.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:48:12.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:48:12.104 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:12.104 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:48:12.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:48:12.203 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T13:48:12.203 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:12.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:48:12.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:48:12.297 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:12.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:48:12.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:48:12.391 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:12.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:48:12.420 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:48:12.490 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:12.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:48:12.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:48:12.581 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T13:48:12.581 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:12.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:48:12.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:48:12.675 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:12.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:48:12.703 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:48:12.769 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:12.769 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:48:12.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:48:12.868 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:12.868 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:48:12.893 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:48:12.959 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T13:48:13.056 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T13:48:13.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-09-13T13:48:13.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-09-13T13:48:13.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-09-13T13:48:13.063 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-09-13T13:48:13.090 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T13:48:13.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:48:13.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:48:13.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:48:13.134 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:48:13.160 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T13:48:13.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:48:13.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:48:13.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:48:13.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:48:13.240 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T13:48:13.241 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T13:48:13.241 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903513', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 4159, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '682/13280', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiZtazMfvLQKL+wFlJOffMbO0/uPP8jSwRCNXFZFASvnEXYDlLFdK3SWEJ7kBLupPB+ZRnzGuqZ9MyvqDrr8aw=', 'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC2wGt7rrPVKkuCK5hKUkN3SFRpUUtXtyP3pEsFzy17etv+MT0fpbkRoNkTJK2io/siRUlJXtogbxBu8hCYdvo=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLi6bw06U9TfThmHoPm/FJp9a2Q/r1TzSwY4TemdRDkkKRhWZmmpBVhh3HGeiaY1XYkod/6nniKlcWHAZOnuEk=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOejw+wAj2h+EGiSnWmxEB9kh+1PjHpuZR4oSPj+Xdtuc1VDjFPGI4k54OQOr6FDl7EDyaJHjfzJxPjXxKTchpE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['ceph config set mgr mgr/balancer/log_level debug', 'ceph balancer status', 'ceph osd set-require-min-compat-client reef', 'ceph balancer mode upmap-read', 'ceph balancer on', 'ceph balancer status', 'ceph balancer status detail']}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:43:51', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T13:48:13.242 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:48:13.263 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:48:13.267 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:48:13.271 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:48:13.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:48:13.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:48:13.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:48:13.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:48:13.353 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:13.353 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:48:13.402 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:48:13.402 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-13 13:48:06.925094484 +0000 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-13 13:45:26.152499535 +0000 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-13 13:45:26.152499535 +0000 2024-09-13T13:48:13.458 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-13 13:45:26.152499535 +0000 2024-09-13T13:48:13.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:48:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-13T13:48:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-13T13:48:13.523 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000204176 s, 2.5 MB/s 2024-09-13T13:48:13.524 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:48:13.582 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-09-13T13:48:13.638 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-13 13:48:06.925094484 +0000 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-13 13:45:26.481491125 +0000 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-13 13:45:26.481491125 +0000 2024-09-13T13:48:13.639 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-13 13:45:26.481491125 +0000 2024-09-13T13:48:13.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:48:13.704 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-13T13:48:13.704 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-13T13:48:13.704 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000217518 s, 2.4 MB/s 2024-09-13T13:48:13.705 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:48:13.762 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-09-13T13:48:13.818 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:48:13.818 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:13.818 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-13 13:48:06.925094484 +0000 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-13 13:45:26.818482509 +0000 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-13 13:45:26.818482509 +0000 2024-09-13T13:48:13.819 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-13 13:45:26.818482509 +0000 2024-09-13T13:48:13.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:48:13.882 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-13T13:48:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-13T13:48:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000184466 s, 2.8 MB/s 2024-09-13T13:48:13.884 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:48:13.941 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-13 13:48:06.926094458 +0000 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-13 13:45:27.166473613 +0000 2024-09-13T13:48:13.997 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-13 13:45:27.166473613 +0000 2024-09-13T13:48:13.998 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-13 13:45:27.166473613 +0000 2024-09-13T13:48:13.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:48:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-13T13:48:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-13T13:48:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000155684 s, 3.3 MB/s 2024-09-13T13:48:14.065 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:48:14.122 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:14.122 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:14.122 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:48:14.141 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:48:14.141 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 13:48:07.718861064 +0000 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 13:45:24.739995650 +0000 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 13:45:24.739995650 +0000 2024-09-13T13:48:14.200 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 13:45:24.739995650 +0000 2024-09-13T13:48:14.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:48:14.271 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T13:48:14.271 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T13:48:14.272 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.00022082 s, 2.3 MB/s 2024-09-13T13:48:14.273 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:48:14.333 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-09-13T13:48:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:48:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 13:48:07.718861064 +0000 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 13:45:25.058987389 +0000 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 13:45:25.058987389 +0000 2024-09-13T13:48:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 13:45:25.058987389 +0000 2024-09-13T13:48:14.392 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:48:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T13:48:14.459 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T13:48:14.459 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000238914 s, 2.1 MB/s 2024-09-13T13:48:14.460 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:48:14.519 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 13:48:07.719861038 +0000 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 13:45:25.396978636 +0000 2024-09-13T13:48:14.577 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 13:45:25.396978636 +0000 2024-09-13T13:48:14.578 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 13:45:25.396978636 +0000 2024-09-13T13:48:14.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:48:14.643 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T13:48:14.643 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T13:48:14.643 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000234497 s, 2.2 MB/s 2024-09-13T13:48:14.644 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:48:14.703 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-13 13:48:07.719861038 +0000 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-13 13:45:25.728970039 +0000 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-13 13:45:25.728970039 +0000 2024-09-13T13:48:14.760 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-13 13:45:25.728970039 +0000 2024-09-13T13:48:14.761 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:48:14.826 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-13T13:48:14.826 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-13T13:48:14.826 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000171771 s, 3.0 MB/s 2024-09-13T13:48:14.827 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:48:14.884 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:14.885 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:14.885 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:48:14.904 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:48:14.904 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-09-13T13:48:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:48:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:14.964 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T13:48:14.964 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:14.964 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:14.964 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 13:48:05.306959959 +0000 2024-09-13T13:48:14.964 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 13:45:26.657177565 +0000 2024-09-13T13:48:14.965 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 13:45:26.657177565 +0000 2024-09-13T13:48:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 13:45:26.657177565 +0000 2024-09-13T13:48:14.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:48:15.031 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T13:48:15.031 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T13:48:15.031 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00025366 s, 2.0 MB/s 2024-09-13T13:48:15.032 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:48:15.091 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-09-13T13:48:15.147 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 13:48:05.306959959 +0000 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 13:45:26.965175907 +0000 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 13:45:26.965175907 +0000 2024-09-13T13:48:15.148 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 13:45:26.965175907 +0000 2024-09-13T13:48:15.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:48:15.212 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T13:48:15.213 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T13:48:15.213 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000193948 s, 2.6 MB/s 2024-09-13T13:48:15.214 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:48:15.271 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-09-13T13:48:15.327 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:48:15.327 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:15.327 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T13:48:15.327 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:15.327 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:15.328 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 13:48:05.306959959 +0000 2024-09-13T13:48:15.328 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 13:45:27.286174179 +0000 2024-09-13T13:48:15.328 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 13:45:27.286174179 +0000 2024-09-13T13:48:15.328 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 13:45:27.286174179 +0000 2024-09-13T13:48:15.328 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:48:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T13:48:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T13:48:15.392 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000194235 s, 2.6 MB/s 2024-09-13T13:48:15.394 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:48:15.450 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-09-13T13:48:15.506 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:48:15.506 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:15.506 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T13:48:15.506 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:15.507 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:15.507 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-13 13:48:05.307959954 +0000 2024-09-13T13:48:15.507 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-13 13:45:27.605172462 +0000 2024-09-13T13:48:15.507 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-13 13:45:27.605172462 +0000 2024-09-13T13:48:15.507 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-13 13:45:27.605172462 +0000 2024-09-13T13:48:15.507 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:48:15.570 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-13T13:48:15.570 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-13T13:48:15.570 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000202517 s, 2.5 MB/s 2024-09-13T13:48:15.571 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:48:15.628 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:15.629 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:15.629 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:48:15.649 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:48:15.649 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-09-13T13:48:15.708 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-13 13:48:03.599179031 +0000 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-13 13:45:32.977013376 +0000 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-13 13:45:32.977013376 +0000 2024-09-13T13:48:15.709 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-13 13:45:32.977013376 +0000 2024-09-13T13:48:15.709 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:48:15.777 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-13T13:48:15.777 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-13T13:48:15.777 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000258915 s, 2.0 MB/s 2024-09-13T13:48:15.779 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:48:15.838 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-09-13T13:48:15.896 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-13 13:48:03.599179031 +0000 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-13 13:45:33.306007189 +0000 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-13 13:45:33.306007189 +0000 2024-09-13T13:48:15.897 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-13 13:45:33.306007189 +0000 2024-09-13T13:48:15.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:48:15.963 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-13T13:48:15.963 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-13T13:48:15.963 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000187375 s, 2.7 MB/s 2024-09-13T13:48:15.965 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:48:16.024 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-09-13T13:48:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:48:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:16.082 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T13:48:16.082 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:16.082 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:16.083 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-13 13:48:03.599179031 +0000 2024-09-13T13:48:16.083 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-13 13:45:33.648000757 +0000 2024-09-13T13:48:16.083 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-13 13:45:33.648000757 +0000 2024-09-13T13:48:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-13 13:45:33.648000757 +0000 2024-09-13T13:48:16.083 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:48:16.148 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-13T13:48:16.148 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-13T13:48:16.148 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000238305 s, 2.1 MB/s 2024-09-13T13:48:16.150 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:48:16.207 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-09-13T13:48:16.263 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:48:16.263 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:48:16.263 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-13T13:48:16.263 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:48:16.263 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:48:16.264 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-09-13 13:48:03.600179013 +0000 2024-09-13T13:48:16.264 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-09-13 13:45:33.964994796 +0000 2024-09-13T13:48:16.264 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-09-13 13:45:33.964994796 +0000 2024-09-13T13:48:16.264 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-09-13 13:45:33.964994796 +0000 2024-09-13T13:48:16.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:48:16.328 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-09-13T13:48:16.328 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-09-13T13:48:16.328 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000224375 s, 2.3 MB/s 2024-09-13T13:48:16.330 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:48:16.386 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:16.387 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T13:48:16.387 INFO:tasks.ceph:Generating config... 2024-09-13T13:48:16.388 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-13T13:48:16.388 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T13:48:16.388 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T13:48:16.388 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] osd beacon report interval = 30 2024-09-13T13:48:16.389 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd map cache size = 1 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd max scrubs = 3 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T13:48:16.390 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T13:48:16.391 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T13:48:16.391 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2024-09-13T13:48:16.391 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-13T13:48:16.391 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-13T13:48:16.391 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-13T13:48:16.391 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] ms type = async 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-13T13:48:16.392 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-13T13:48:16.392 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T13:48:16.393 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T13:48:16.393 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T13:48:16.393 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T13:48:16.393 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-13T13:48:16.393 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T13:48:16.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T13:48:16.454 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T13:48:16.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T13:48:16.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:48:16.532 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.100'), ('mon.c', '172.21.15.122')] 2024-09-13T13:48:16.532 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.100,172.21.15.122', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-13T13:48:16.533 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:16.533 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T13:48:16.589 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.100 --add c 172.21.15.122 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:fsid cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-09-13T13:48:16.679887+0000 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-09-13T13:48:16.679887+0000 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-09-13T13:48:16.680 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-09-13T13:48:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-09-13T13:48:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.b 2024-09-13T13:48:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.c 2024-09-13T13:48:16.681 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T13:48:16.682 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T13:48:16.698 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cd1b3cf6-d610-4c64-94c3-504da850dd35... 2024-09-13T13:48:16.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:48:16.741 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:48:16.743 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:48:16.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.783 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-09-13T13:48:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:48:16.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.100,172.21.15.122" 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon client directed command retry = 5 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: mon election default strategy = 1 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: ms bind msgr1 = False 2024-09-13T13:48:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: ms bind msgr2 = True 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject delay max = 1 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject delay probability = 0.005 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject delay type = osd 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject internal delays = 0.002 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms inject socket failures = 2500 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: ms type = async 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: osd_async_recovery_min_cost = 1 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_min_size = 2 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_size = 2 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.787 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 120 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-09-13T13:48:16.788 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-09-13T13:48:16.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: debug monc = 20 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore write v2 = True 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore zero block detection = True 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 20 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 20 2024-09-13T13:48:16.790 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 10 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd beacon report interval = 30 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd blocked scrub grace period = 3600 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify cached snaps = True 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify missing on start = True 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd inject bad map crc probability = 0.1 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd map cache size = 1 2024-09-13T13:48:16.791 INFO:teuthology.orchestra.run.smithi027.stdout: osd max backfills = 6 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd max markdown count = 1000 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd max scrubs = 3 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock override recovery settings = True 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd memory target = 1610612736 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd op queue = debug_random 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd op queue cut off = debug_random 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub during recovery = False 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub min interval = 60 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd shutdown pgref assert = True 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-09-13T13:48:16.792 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-13T13:48:16.793 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:48:16.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon min osdmap epochs = 50 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune interval = 2 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune min = 15 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon scrub interval = 300 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: paxos service trim min = 10 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:48:16.795 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-09-13T13:48:16.796 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.797 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:48:16.798 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-09-13T13:48:16.799 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.800 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "172.21.15.27,172.21.15.100,172.21.15.122" 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: mon client directed command retry = 5 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: mon election default strategy = 1 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms bind msgr1 = False 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms bind msgr2 = True 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay max = 1 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay probability = 0.005 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject delay type = osd 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject internal delays = 0.002 2024-09-13T13:48:16.801 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject socket failures = 2500 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: ms type = async 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd_async_recovery_min_cost = 1 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_min_size = 2 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_size = 2 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-09-13T13:48:16.802 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 120 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-09-13T13:48:16.803 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-09-13T13:48:16.804 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore write v2 = True 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore zero block detection = True 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:48:16.805 INFO:teuthology.orchestra.run.smithi159.stdout: osd beacon report interval = 30 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd blocked scrub grace period = 3600 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify cached snaps = True 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify missing on start = True 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd inject bad map crc probability = 0.1 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd map cache size = 1 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd max backfills = 6 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd max markdown count = 1000 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd max scrubs = 3 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock override recovery settings = True 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd memory target = 1610612736 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue = debug_random 2024-09-13T13:48:16.806 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue cut off = debug_random 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub during recovery = False 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub min interval = 60 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: osd shutdown pgref assert = True 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-09-13T13:48:16.807 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.808 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon min osdmap epochs = 50 2024-09-13T13:48:16.809 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune interval = 2 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune min = 15 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: mon scrub interval = 300 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: paxos service trim min = 10 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-09-13T13:48:16.810 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-13T13:48:16.811 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-13T13:48:16.812 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-13T13:48:16.813 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.27,172.21.15.100,172.21.15.122" 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: mon election default strategy = 1 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms bind msgr1 = False 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms bind msgr2 = True 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject delay max = 1 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject delay probability = 0.005 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject delay type = osd 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject internal delays = 0.002 2024-09-13T13:48:16.814 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 2500 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: ms type = async 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd_async_recovery_min_cost = 1 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_min_size = 2 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_size = 2 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 120 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.815 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.816 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 20 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore write v2 = True 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore zero block detection = True 2024-09-13T13:48:16.817 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 20 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 20 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 10 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd beacon report interval = 30 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd blocked scrub grace period = 3600 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify cached snaps = True 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify missing on start = True 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd inject bad map crc probability = 0.1 2024-09-13T13:48:16.818 INFO:teuthology.orchestra.run.smithi122.stdout: osd map cache size = 1 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd max backfills = 6 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd max markdown count = 1000 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd max scrubs = 3 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock override recovery settings = True 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd memory target = 1610612736 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue = debug_random 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue cut off = debug_random 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub during recovery = False 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub min interval = 60 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: osd shutdown pgref assert = True 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.819 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-13T13:48:16.820 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:48:16.821 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon min osdmap epochs = 50 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune interval = 2 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune min = 15 2024-09-13T13:48:16.822 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub interval = 300 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: paxos service trim min = 10 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-09-13T13:48:16.823 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-09-13T13:48:16.824 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:48:16.825 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-09-13T13:48:16.826 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.27,172.21.15.100,172.21.15.122" 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: mon client directed command retry = 5 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: mon election default strategy = 1 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms bind msgr1 = False 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms bind msgr2 = True 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject delay max = 1 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject delay probability = 0.005 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject delay type = osd 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject internal delays = 0.002 2024-09-13T13:48:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject socket failures = 2500 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: ms type = async 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd_async_recovery_min_cost = 1 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_min_size = 2 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_size = 2 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 120 2024-09-13T13:48:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug trim objects = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.829 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: debug monc = 20 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-09-13T13:48:16.830 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore write v2 = True 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore zero block detection = True 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 20 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 20 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 10 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-09-13T13:48:16.831 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd beacon report interval = 30 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd blocked scrub grace period = 3600 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify cached snaps = True 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify missing on start = True 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd inject bad map crc probability = 0.1 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd map cache size = 1 2024-09-13T13:48:16.832 INFO:teuthology.orchestra.run.smithi100.stdout: osd max backfills = 6 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd max markdown count = 1000 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd max scrubs = 3 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock override recovery settings = True 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd memory target = 1610612736 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue = debug_random 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue cut off = debug_random 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub during recovery = False 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub min interval = 60 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: osd shutdown pgref assert = True 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.833 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-13T13:48:16.834 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:48:16.835 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon min osdmap epochs = 50 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune interval = 2 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune min = 15 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub interval = 300 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: paxos service trim min = 10 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-13T13:48:16.836 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-09-13T13:48:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-09-13T13:48:16.840 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T13:48:16.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T13:48:16.936 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T13:48:16.936 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:16.936 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T13:48:16.952 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:16.952 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T13:48:17.008 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:48:17.008 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:17.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:48:17.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:48:17.083 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:48:17.083 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:17.139 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:48:17.139 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:17.139 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:48:17.140 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:48:17.181 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:48:17.181 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:17.239 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:48:17.239 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:17.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:48:17.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:48:17.283 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:48:17.283 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:17.341 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:48:17.342 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:17.342 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:48:17.342 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:48:17.383 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:48:17.383 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:17.440 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T13:48:17.440 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T13:48:17.441 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T13:48:17.494 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T13:48:17.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T13:48:17.566 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T13:48:17.569 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T13:48:17.569 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T13:48:17.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T13:48:17.614 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T13:48:17.628 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T13:48:17.628 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T13:48:17.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T13:48:17.658 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:17.658 INFO:tasks.ceph:role: osd.0 2024-09-13T13:48:17.659 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:48:17.659 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:48:17.731 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:17.732 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:17.733 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:17.733 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:18.220 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-13T13:48:18.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-13T13:48:18.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T13:48:18.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T13:48:18.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T13:48:18.467 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:18.467 INFO:tasks.ceph:role: osd.4 2024-09-13T13:48:18.467 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:48:18.467 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:18.543 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:18.991 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-13T13:48:18.993 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-13T13:48:18.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T13:48:19.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T13:48:19.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-13T13:48:19.189 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:19.190 INFO:tasks.ceph:role: osd.8 2024-09-13T13:48:19.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:48:19.190 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:19.265 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:19.266 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:19.266 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:19.266 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:19.756 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-13T13:48:19.758 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-13T13:48:19.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-13T13:48:19.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-13T13:48:19.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-13T13:48:19.994 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:19.994 INFO:tasks.ceph:role: osd.12 2024-09-13T13:48:19.994 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:48:19.994 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:20.069 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:20.070 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:20.070 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:20.566 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-13T13:48:20.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-13T13:48:20.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-13T13:48:20.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-13T13:48:20.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:20.839+0000 7f2ade41c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T13:48:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:20.839+0000 7f2ade41c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T13:48:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:20.840+0000 7f2ade41c640 -1 bdev(0x563020839800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:20.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:20.840+0000 7f2ade41c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T13:48:23.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T13:48:23.778 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:23.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:23.869+0000 7feaf8011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T13:48:23.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:23.869+0000 7feaf8011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T13:48:23.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:23.869+0000 7feaf8011640 -1 bdev(0x55d0419f9800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:23.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:23.869+0000 7feaf8011640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T13:48:27.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:48:27.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:27.153+0000 7f9908c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-13T13:48:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:27.153+0000 7f9908c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-13T13:48:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:27.153+0000 7f9908c0d640 -1 bdev(0x55ab3a719800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:27.153+0000 7f9908c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-13T13:48:30.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-13T13:48:30.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:30.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:30.453+0000 7ff39c8e2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-13T13:48:30.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:30.453+0000 7ff39c8e2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-13T13:48:30.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:30.453+0000 7ff39c8e2640 -1 bdev(0x55ee3c5db800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:30.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:48:30.453+0000 7ff39c8e2640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-13T13:48:33.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-13T13:48:33.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T13:48:33.675 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:33.676 INFO:tasks.ceph:role: osd.1 2024-09-13T13:48:33.676 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:48:33.676 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:33.752 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:33.753 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:33.753 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:34.229 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T13:48:34.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T13:48:34.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-13T13:48:34.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T13:48:34.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T13:48:34.456 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:34.456 INFO:tasks.ceph:role: osd.5 2024-09-13T13:48:34.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:48:34.457 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:48:34.530 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:34.530 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:34.531 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:34.986 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T13:48:34.988 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T13:48:34.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T13:48:35.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T13:48:35.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-13T13:48:35.196 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:35.197 INFO:tasks.ceph:role: osd.9 2024-09-13T13:48:35.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:48:35.197 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:35.268 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:35.269 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:35.269 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:35.745 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T13:48:35.746 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T13:48:35.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-13T13:48:35.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-13T13:48:35.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-13T13:48:35.944 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:35.944 INFO:tasks.ceph:role: osd.13 2024-09-13T13:48:35.944 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:48:35.944 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T13:48:36.016 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:36.017 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:36.518 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-13T13:48:36.520 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-13T13:48:36.521 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-13T13:48:36.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-13T13:48:36.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:36.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:36.770+0000 7f551ffdb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T13:48:36.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:36.771+0000 7f551ffdb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T13:48:36.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:36.771+0000 7f551ffdb640 -1 bdev(0x5571be24f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:36.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:36.771+0000 7f551ffdb640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T13:48:39.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T13:48:39.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:39.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:39.795+0000 7f89b6c0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T13:48:39.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:39.795+0000 7f89b6c0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T13:48:39.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:39.795+0000 7f89b6c0f640 -1 bdev(0x55a2b1279800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:39.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:39.795+0000 7f89b6c0f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T13:48:42.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T13:48:42.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:43.072 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:43.070+0000 7f43f7c2a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-13T13:48:43.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:43.071+0000 7f43f7c2a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-13T13:48:43.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:43.071+0000 7f43f7c2a640 -1 bdev(0x55a80e0ab800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:43.073 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:43.071+0000 7f43f7c2a640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-13T13:48:46.248 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-13T13:48:46.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:46.364 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:46.362+0000 7f4ea820d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-13T13:48:46.364 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:46.362+0000 7f4ea820d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-13T13:48:46.364 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:46.362+0000 7f4ea820d640 -1 bdev(0x55f7ff63f800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:46.364 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-13T13:48:46.363+0000 7f4ea820d640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-13T13:48:49.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-13T13:48:49.298 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T13:48:49.329 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:49.329 INFO:tasks.ceph:role: osd.2 2024-09-13T13:48:49.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:48:49.329 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:48:49.405 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:49.405 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:49.405 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:49.405 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:49.405 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:49.406 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:49.406 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:49.406 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:49.406 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:49.406 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:49.910 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T13:48:49.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T13:48:49.913 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-13T13:48:50.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T13:48:50.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T13:48:50.153 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:50.153 INFO:tasks.ceph:role: osd.6 2024-09-13T13:48:50.153 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:48:50.153 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:48:50.225 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:50.225 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:50.225 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:50.225 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:50.226 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:50.667 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T13:48:50.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T13:48:50.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-13T13:48:50.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T13:48:50.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-13T13:48:50.853 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:50.854 INFO:tasks.ceph:role: osd.10 2024-09-13T13:48:50.854 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:48:50.854 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:48:50.925 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:50.926 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:51.402 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T13:48:51.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T13:48:51.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-13T13:48:51.543 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-13T13:48:51.573 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-13T13:48:51.638 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T13:48:51.638 INFO:tasks.ceph:role: osd.14 2024-09-13T13:48:51.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:48:51.639 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:48:51.712 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:48:51.713 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:48:51.713 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:48:52.215 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-13T13:48:52.217 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-13T13:48:52.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-13T13:48:52.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-13T13:48:52.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:52.488 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:52.487+0000 7f272a615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T13:48:52.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:52.487+0000 7f272a615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T13:48:52.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:52.487+0000 7f272a615640 -1 bdev(0x563b560cd800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:52.489 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:52.488+0000 7f272a615640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T13:48:55.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T13:48:55.431 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:55.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:55.522+0000 7fe3f5d2b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T13:48:55.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:55.522+0000 7fe3f5d2b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T13:48:55.523 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:55.522+0000 7fe3f5d2b640 -1 bdev(0x55952d379800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:55.524 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:55.522+0000 7fe3f5d2b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T13:48:58.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T13:48:58.466 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:48:58.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:58.557+0000 7f6dc86ef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-13T13:48:58.558 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:58.557+0000 7f6dc86ef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-13T13:48:58.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:58.557+0000 7f6dc86ef640 -1 bdev(0x5614a8633800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-13T13:48:58.559 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:48:58.557+0000 7f6dc86ef640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-13T13:49:01.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-13T13:49:01.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:01.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:49:01.824+0000 7fc5678eb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-13T13:49:01.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:49:01.825+0000 7fc5678eb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-13T13:49:01.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:49:01.825+0000 7fc5678eb640 -1 bdev(0x563dfff2f800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-13T13:49:01.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-13T13:49:01.825+0000 7fc5678eb640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-13T13:49:04.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-13T13:49:04.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T13:49:04.799 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T13:49:04.799 INFO:tasks.ceph:role: osd.3 2024-09-13T13:49:04.799 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:49:04.799 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:49:04.876 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:49:04.877 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:49:05.396 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-13T13:49:05.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-13T13:49:05.398 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T13:49:05.540 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T13:49:05.573 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T13:49:05.639 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T13:49:05.639 INFO:tasks.ceph:role: osd.7 2024-09-13T13:49:05.639 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:49:05.639 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:49:05.710 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:49:05.710 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:49:05.711 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:49:06.154 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-13T13:49:06.156 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-13T13:49:06.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-13T13:49:06.241 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T13:49:06.310 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-13T13:49:06.374 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T13:49:06.374 INFO:tasks.ceph:role: osd.11 2024-09-13T13:49:06.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:49:06.374 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:49:06.447 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:49:06.447 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:49:06.448 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:49:06.950 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-13T13:49:06.952 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-13T13:49:06.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-13T13:49:07.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-13T13:49:07.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-13T13:49:07.169 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T13:49:07.169 INFO:tasks.ceph:role: osd.15 2024-09-13T13:49:07.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:49:07.169 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:49:07.241 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:49:07.242 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:49:07.242 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:49:07.242 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:49:07.242 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:49:07.763 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-09-13T13:49:07.765 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-09-13T13:49:07.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-13T13:49:07.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-13T13:49:07.923 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:08.033 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:08.028+0000 7f54ec40b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T13:49:08.033 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:08.028+0000 7f54ec40b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T13:49:08.033 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:08.028+0000 7f54ec40b640 -1 bdev(0x55bc4519b800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T13:49:08.033 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:08.028+0000 7f54ec40b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T13:49:11.191 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T13:49:11.221 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:11.313 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:11.311+0000 7f882222e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T13:49:11.313 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:11.311+0000 7f882222e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T13:49:11.313 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:11.311+0000 7f882222e640 -1 bdev(0x55c82cc6d800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-13T13:49:11.314 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:11.311+0000 7f882222e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T13:49:14.476 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T13:49:14.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:14.596 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:14.594+0000 7f7045018640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-13T13:49:14.596 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:14.594+0000 7f7045018640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-13T13:49:14.596 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:14.594+0000 7f7045018640 -1 bdev(0x5617acb4b800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-13T13:49:14.596 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:14.594+0000 7f7045018640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-13T13:49:17.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-13T13:49:17.535 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:17.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:17.625+0000 7f0fe0f58640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-13T13:49:17.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:17.625+0000 7f0fe0f58640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-13T13:49:17.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:17.625+0000 7f0fe0f58640 -1 bdev(0x55ae44229800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-13T13:49:17.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-09-13T13:49:17.625+0000 7f0fe0f58640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-13T13:49:20.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-13T13:49:20.833 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T13:49:20.833 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:20.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T13:49:20.865 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:20.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T13:49:20.935 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:20.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T13:49:21.005 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:21.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-13T13:49:21.073 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:21.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-13T13:49:21.141 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:21.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T13:49:21.172 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:21.172 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T13:49:21.243 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:21.243 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-13T13:49:21.311 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:21.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-13T13:49:21.377 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:21.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T13:49:21.408 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:21.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T13:49:21.478 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:21.478 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-13T13:49:21.548 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:21.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-13T13:49:21.616 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T13:49:21.647 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.647 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T13:49:21.716 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.716 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T13:49:21.784 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.784 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-13T13:49:21.851 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.851 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-13T13:49:21.917 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:21.918 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T13:49:21.975 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T13:49:21.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T13:49:21.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T13:49:21.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T13:49:22.000 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.y] 2024-09-13T13:49:22.001 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQChQuRm0vF5HRAAB5vO6YfE/gr79qPkaZgnZQ== 2024-09-13T13:49:22.001 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-09-13T13:49:22.001 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCkQuRm1UoiMhAAjS7Yo+G6AhCkUIttaI/AmA== 2024-09-13T13:49:22.001 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-09-13T13:49:22.001 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCnQuRmgwPmMxAAM3/egCoYQMFclyItQ2hV1A== 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.8] 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCrQuRmITo6CRAAa5K6Eh0iOgCUVpFsJ8bNag== 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.12] 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCuQuRmCugXGxAApIRX8EsFjaprczxgwraPmw== 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC0QuRmgK71LRAAu+UqBUJ9RgBT1zdBzA1WRg== 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC3QuRmEv5uLxAAIcGRmWIaqScQcCKVOQ8zyg== 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.9] 2024-09-13T13:49:22.002 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC7QuRmN4ZPBBAA4G+nJgP7iKbP7MoUihK8mA== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.13] 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC+QuRm8nuxFRAASA/f2hLzVqLNuA2Be8nAFQ== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDEQuRmAkUdHRAAYpy0CbcGm3DBiwRlRuaVUA== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.6] 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDHQuRmO7kwHxAAeWGD7Jj+4ZooEEhhbLw0iQ== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.10] 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDKQuRm/9ZGIRAA89NwxWgaF4jhULLC0Dfy3Q== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.14] 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDNQuRm+Mo4MRAA7iK0vtYbUqmrWjI9ukrgew== 2024-09-13T13:49:22.003 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQChQuRmjkm4IRAAGG16uTyXyYMb4XeOyaVqhw== 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDUQuRmxMjJARAAXhUD4jkloNWOL8QmbN+t4g== 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.7] 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDXQuRmuOGpEhAAwVsG4XGC9qIdpvShAu1idQ== 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.11] 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDaQuRmTTWLIxAAdPHKr6fnt1T23p612fFTjA== 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.15] 2024-09-13T13:49:22.004 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDdQuRmX/tiJRAA7RTWvNOEWfV5YoMH+DmSMA== 2024-09-13T13:49:22.005 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-09-13T13:49:22.005 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQChQuRmnDWhJBAARza3pK+ewBmH8oMv7UI4kg== 2024-09-13T13:49:22.006 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-09-13T13:49:22.006 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQChQuRm0vF5HRAAB5vO6YfE/gr79qPkaZgnZQ== 2024-09-13T13:49:22.006 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCkQuRm1UoiMhAAjS7Yo+G6AhCkUIttaI/AmA== 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCnQuRmgwPmMxAAM3/egCoYQMFclyItQ2hV1A== 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.8] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCrQuRmITo6CRAAa5K6Eh0iOgCUVpFsJ8bNag== 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.12] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCuQuRmCugXGxAApIRX8EsFjaprczxgwraPmw== 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC0QuRmgK71LRAAu+UqBUJ9RgBT1zdBzA1WRg== 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2024-09-13T13:49:22.007 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC3QuRmEv5uLxAAIcGRmWIaqScQcCKVOQ8zyg== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.9] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC7QuRmN4ZPBBAA4G+nJgP7iKbP7MoUihK8mA== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.13] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQC+QuRm8nuxFRAASA/f2hLzVqLNuA2Be8nAFQ== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDEQuRmAkUdHRAAYpy0CbcGm3DBiwRlRuaVUA== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDHQuRmO7kwHxAAeWGD7Jj+4ZooEEhhbLw0iQ== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.10] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDKQuRm/9ZGIRAA89NwxWgaF4jhULLC0Dfy3Q== 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.14] 2024-09-13T13:49:22.008 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDNQuRm+Mo4MRAA7iK0vtYbUqmrWjI9ukrgew== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQChQuRmjkm4IRAAGG16uTyXyYMb4XeOyaVqhw== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDUQuRmxMjJARAAXhUD4jkloNWOL8QmbN+t4g== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDXQuRmuOGpEhAAwVsG4XGC9qIdpvShAu1idQ== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.11] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDaQuRmTTWLIxAAdPHKr6fnt1T23p612fFTjA== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.15] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDdQuRmX/tiJRAA7RTWvNOEWfV5YoMH+DmSMA== 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-09-13T13:49:22.009 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQChQuRmnDWhJBAARza3pK+ewBmH8oMv7UI4kg== 2024-09-13T13:49:22.010 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.y] 2024-09-13T13:49:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQChQuRm0vF5HRAAB5vO6YfE/gr79qPkaZgnZQ== 2024-09-13T13:49:22.011 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-09-13T13:49:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCkQuRm1UoiMhAAjS7Yo+G6AhCkUIttaI/AmA== 2024-09-13T13:49:22.011 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2024-09-13T13:49:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCnQuRmgwPmMxAAM3/egCoYQMFclyItQ2hV1A== 2024-09-13T13:49:22.012 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.8] 2024-09-13T13:49:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCrQuRmITo6CRAAa5K6Eh0iOgCUVpFsJ8bNag== 2024-09-13T13:49:22.012 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.12] 2024-09-13T13:49:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQCuQuRmCugXGxAApIRX8EsFjaprczxgwraPmw== 2024-09-13T13:49:22.012 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQC0QuRmgK71LRAAu+UqBUJ9RgBT1zdBzA1WRg== 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQC3QuRmEv5uLxAAIcGRmWIaqScQcCKVOQ8zyg== 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.9] 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQC7QuRmN4ZPBBAA4G+nJgP7iKbP7MoUihK8mA== 2024-09-13T13:49:22.013 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.13] 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQC+QuRm8nuxFRAASA/f2hLzVqLNuA2Be8nAFQ== 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDEQuRmAkUdHRAAYpy0CbcGm3DBiwRlRuaVUA== 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.6] 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDHQuRmO7kwHxAAeWGD7Jj+4ZooEEhhbLw0iQ== 2024-09-13T13:49:22.014 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.10] 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDKQuRm/9ZGIRAA89NwxWgaF4jhULLC0Dfy3Q== 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.14] 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDNQuRm+Mo4MRAA7iK0vtYbUqmrWjI9ukrgew== 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQChQuRmjkm4IRAAGG16uTyXyYMb4XeOyaVqhw== 2024-09-13T13:49:22.015 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDUQuRmxMjJARAAXhUD4jkloNWOL8QmbN+t4g== 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.7] 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDXQuRmuOGpEhAAwVsG4XGC9qIdpvShAu1idQ== 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.11] 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDaQuRmTTWLIxAAdPHKr6fnt1T23p612fFTjA== 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.15] 2024-09-13T13:49:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDdQuRmX/tiJRAA7RTWvNOEWfV5YoMH+DmSMA== 2024-09-13T13:49:22.017 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-09-13T13:49:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQChQuRmnDWhJBAARza3pK+ewBmH8oMv7UI4kg== 2024-09-13T13:49:22.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.173 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.229 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.282 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.338 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.453 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.511 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:49:22.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.952 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:22.954 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:23.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:23.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:23.010 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:49:23.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T13:49:23.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T13:49:23.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T13:49:23.123 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T13:49:23.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T13:49:23.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:49:23.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T13:49:23.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T13:49:23.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:49:23.394 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T13:49:23.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T13:49:23.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:49:23.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T13:49:23.564 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:23.567 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:23.569 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:49:23.620 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T13:49:23.620 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T13:49:23.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T13:49:23.623 INFO:tasks.ceph.mon.a:Started 2024-09-13T13:49:23.623 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T13:49:23.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T13:49:23.626 INFO:tasks.ceph.mon.b:Started 2024-09-13T13:49:23.626 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T13:49:23.626 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T13:49:23.662 INFO:tasks.ceph.mon.c:Started 2024-09-13T13:49:23.662 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T13:49:23.663 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T13:49:23.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T13:49:23.665 INFO:tasks.ceph.mgr.y:Started 2024-09-13T13:49:23.665 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T13:49:23.665 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T13:49:23.668 INFO:tasks.ceph.mgr.x:Started 2024-09-13T13:49:23.668 DEBUG:tasks.ceph:set 0 configs 2024-09-13T13:49:23.668 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T13:49:23.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T13:49:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-09-13T13:49:34.095 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T13:49:34.095 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T13:49:34.096 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:34.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T13:49:34.122 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:34.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T13:49:34.188 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:34.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-13T13:49:34.254 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-13T13:49:34.255 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-13T13:49:34.320 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:34.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T13:49:34.351 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:34.351 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T13:49:34.422 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:34.422 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-13T13:49:34.489 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-13T13:49:34.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-13T13:49:34.558 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:34.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T13:49:34.589 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:34.589 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T13:49:34.660 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:34.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-13T13:49:34.731 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-13T13:49:34.731 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-13T13:49:34.801 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:34.801 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T13:49:34.826 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:34.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T13:49:34.890 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:34.890 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-13T13:49:34.955 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-09-13T13:49:34.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-13T13:49:35.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 1259d7f4-8ac0-4fdc-8473-9bdafc036502 0 2024-09-13T13:49:35.246 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-09-13T13:49:35.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 9035b2e5-ff53-4675-83b8-bf76ab2d3735 1 2024-09-13T13:49:35.443 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-09-13T13:49:35.456 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 497bd315-4b6e-4f41-b0de-488a942ac4d9 2 2024-09-13T13:49:35.647 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-09-13T13:49:35.659 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 024edf52-c84f-4d52-b825-c203ed5aecb6 3 2024-09-13T13:49:35.841 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-09-13T13:49:35.853 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new c440743d-dac9-4c75-b6b3-94360a48f75c 4 2024-09-13T13:49:36.041 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-09-13T13:49:36.052 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new d8203cad-54c8-43f1-af73-ec199c8a41c9 5 2024-09-13T13:49:36.240 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-09-13T13:49:36.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new d5e9b934-6b1e-4f27-838c-6529f69a5b65 6 2024-09-13T13:49:36.434 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-09-13T13:49:36.445 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 46d21c0b-bd67-45c9-8bcc-9e76edd03769 7 2024-09-13T13:49:36.633 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-09-13T13:49:36.645 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0 8 2024-09-13T13:49:36.831 INFO:teuthology.orchestra.run.smithi159.stdout:8 2024-09-13T13:49:36.843 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d 9 2024-09-13T13:49:37.028 INFO:teuthology.orchestra.run.smithi159.stdout:9 2024-09-13T13:49:37.039 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 417658a1-1270-435c-8e2a-b9e4a8b0dba6 10 2024-09-13T13:49:37.221 INFO:teuthology.orchestra.run.smithi159.stdout:10 2024-09-13T13:49:37.233 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 9191abab-883f-40a1-8176-3c2e74d90c07 11 2024-09-13T13:49:37.414 INFO:teuthology.orchestra.run.smithi159.stdout:11 2024-09-13T13:49:37.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new b9a2390c-9646-489d-b46b-268aa3e73194 12 2024-09-13T13:49:37.607 INFO:teuthology.orchestra.run.smithi159.stdout:12 2024-09-13T13:49:37.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18 13 2024-09-13T13:49:37.801 INFO:teuthology.orchestra.run.smithi159.stdout:13 2024-09-13T13:49:37.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 99ad3153-7823-4dd9-9c09-044eba14bcd5 14 2024-09-13T13:49:37.996 INFO:teuthology.orchestra.run.smithi159.stdout:14 2024-09-13T13:49:38.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 0880b9c1-ac74-4794-b193-f21d0f1a27f6 15 2024-09-13T13:49:38.191 INFO:teuthology.orchestra.run.smithi159.stdout:15 2024-09-13T13:49:38.203 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T13:49:38.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T13:49:38.205 INFO:tasks.ceph.osd.0:Started 2024-09-13T13:49:38.205 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T13:49:38.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T13:49:38.207 INFO:tasks.ceph.osd.4:Started 2024-09-13T13:49:38.207 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-13T13:49:38.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-13T13:49:38.210 INFO:tasks.ceph.osd.8:Started 2024-09-13T13:49:38.210 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-13T13:49:38.210 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-13T13:49:38.212 INFO:tasks.ceph.osd.12:Started 2024-09-13T13:49:38.212 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T13:49:38.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T13:49:38.215 INFO:tasks.ceph.osd.1:Started 2024-09-13T13:49:38.215 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T13:49:38.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T13:49:38.218 INFO:tasks.ceph.osd.5:Started 2024-09-13T13:49:38.218 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-13T13:49:38.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-13T13:49:38.220 INFO:tasks.ceph.osd.9:Started 2024-09-13T13:49:38.220 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-13T13:49:38.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-13T13:49:38.224 INFO:tasks.ceph.osd.13:Started 2024-09-13T13:49:38.224 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T13:49:38.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T13:49:38.227 INFO:tasks.ceph.osd.2:Started 2024-09-13T13:49:38.228 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T13:49:38.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T13:49:38.230 INFO:tasks.ceph.osd.6:Started 2024-09-13T13:49:38.230 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-13T13:49:38.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-13T13:49:38.233 INFO:tasks.ceph.osd.10:Started 2024-09-13T13:49:38.233 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-13T13:49:38.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-13T13:49:38.236 INFO:tasks.ceph.osd.14:Started 2024-09-13T13:49:38.236 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T13:49:38.236 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T13:49:38.239 INFO:tasks.ceph.osd.3:Started 2024-09-13T13:49:38.239 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T13:49:38.239 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T13:49:38.241 INFO:tasks.ceph.osd.7:Started 2024-09-13T13:49:38.241 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-13T13:49:38.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-13T13:49:38.244 INFO:tasks.ceph.osd.11:Started 2024-09-13T13:49:38.244 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-13T13:49:38.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-13T13:49:38.247 INFO:tasks.ceph.osd.15:Started 2024-09-13T13:49:38.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:49:38.464 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:38.464 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:49:38.189589+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T13:49:38.189589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:49:38.475 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T13:49:38.475 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T13:49:39.366 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:39.364+0000 7f1a62d4d640 -1 Falling back to public interface 2024-09-13T13:49:39.382 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-13T13:49:39.381+0000 7f84c70d6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T13:49:39.383 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-09-13T13:49:39.382+0000 7f84c70d6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T13:49:39.419 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:49:39.417+0000 7f3c4460d640 -1 Falling back to public interface 2024-09-13T13:49:39.426 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:39.424+0000 7f9b1abae640 -1 Falling back to public interface 2024-09-13T13:49:39.442 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:49:39.441+0000 7fe10845e640 -1 Falling back to public interface 2024-09-13T13:49:39.649 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:39.647+0000 7faebce15640 -1 Falling back to public interface 2024-09-13T13:49:39.660 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:49:39.659+0000 7f8f6780b640 -1 Falling back to public interface 2024-09-13T13:49:39.676 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:49:39.674+0000 7f500e3dd640 -1 Falling back to public interface 2024-09-13T13:49:39.687 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:49:39.686+0000 7f7c84824640 -1 Falling back to public interface 2024-09-13T13:49:39.691 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:49:39.690+0000 7f2291c15640 -1 Falling back to public interface 2024-09-13T13:49:39.694 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:39.692+0000 7ffb01a0b640 -1 Falling back to public interface 2024-09-13T13:49:39.698 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:49:39.696+0000 7f1f47dc3640 -1 Falling back to public interface 2024-09-13T13:49:39.702 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:39.702+0000 7f26c9639640 -1 Falling back to public interface 2024-09-13T13:49:39.719 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:39.718+0000 7f004023e640 -1 Falling back to public interface 2024-09-13T13:49:39.723 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:39.722+0000 7fc17d22e640 -1 Falling back to public interface 2024-09-13T13:49:39.914 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:39.913+0000 7f587d417640 -1 Falling back to public interface 2024-09-13T13:49:40.086 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T13:49:40.179 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:49:40.178+0000 7fe86a5b2640 -1 Falling back to public interface 2024-09-13T13:49:40.397 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-13T13:49:40.398 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":18,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:49:38.189589+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T13:49:38.189589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:49:40.409 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-13T13:49:42.916 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:42.914+0000 7f1a62d4d640 -1 osd.13 0 log_to_monitors true 2024-09-13T13:49:43.163 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:43.162+0000 7faebce15640 -1 osd.8 0 log_to_monitors true 2024-09-13T13:49:43.273 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:43.271+0000 7f9b1abae640 -1 osd.12 0 log_to_monitors true 2024-09-13T13:49:43.463 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:43.462+0000 7f26c9639640 -1 osd.3 0 log_to_monitors true 2024-09-13T13:49:43.501 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:43.499+0000 7ffb01a0b640 -1 osd.0 0 log_to_monitors true 2024-09-13T13:49:43.526 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:49:43.525+0000 7f2291c15640 -1 osd.10 0 log_to_monitors true 2024-09-13T13:49:43.540 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:43.540+0000 7f004023e640 -1 osd.15 0 log_to_monitors true 2024-09-13T13:49:43.740 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:49:43.739+0000 7f3c4460d640 -1 osd.5 0 log_to_monitors true 2024-09-13T13:49:43.757 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:49:43.756+0000 7fe10845e640 -1 osd.2 0 log_to_monitors true 2024-09-13T13:49:43.790 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:43.789+0000 7fc17d22e640 -1 osd.11 0 log_to_monitors true 2024-09-13T13:49:43.924 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:49:43.923+0000 7f8f6780b640 -1 osd.6 0 log_to_monitors true 2024-09-13T13:49:43.955 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:49:43.954+0000 7fe86a5b2640 -1 osd.7 0 log_to_monitors true 2024-09-13T13:49:43.959 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:43.959+0000 7f587d417640 -1 osd.9 0 log_to_monitors true 2024-09-13T13:49:43.974 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:49:43.973+0000 7f1f47dc3640 -1 osd.4 0 log_to_monitors true 2024-09-13T13:49:43.985 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:49:43.985+0000 7f500e3dd640 -1 osd.1 0 log_to_monitors true 2024-09-13T13:49:44.009 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:49:44.008+0000 7f7c84824640 -1 osd.14 0 log_to_monitors true 2024-09-13T13:49:44.653 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:44.653+0000 7f1a5ecdc640 -1 osd.13 0 waiting for initial osdmap 2024-09-13T13:49:44.654 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:44.653+0000 7f1a526ee640 -1 osd.13 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:44.662 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:44.662+0000 7f1a56afb640 -1 osd.13 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:44.987 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:49:44.987+0000 7f3c3ff1e640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T13:49:44.989 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:49:44.988+0000 7fe1043ed640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T13:49:44.989 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:49:44.988+0000 7f8f6324b640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T13:49:44.990 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:49:44.989+0000 7fe0fbe07640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:44.990 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:49:44.989+0000 7f8f5ac65640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:44.990 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:49:44.989+0000 7f228d4f5640 -1 osd.10 0 waiting for initial osdmap 2024-09-13T13:49:44.991 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:49:44.990+0000 7f2284f10640 -1 osd.10 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:44.992 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:44.991+0000 7f26c55c8640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T13:49:44.992 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:44.992+0000 7f26bcfe2640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:45.000 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:49:44.999+0000 7f228931c640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.000 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:49:45.000+0000 7fe100214640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.002 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:45.001+0000 7f9b16b3d640 -1 osd.12 0 waiting for initial osdmap 2024-09-13T13:49:45.002 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:45.001+0000 7f9b0e557640 -1 osd.12 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:45.003 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:49:45.003+0000 7f8f5f072640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.013 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:49:45.012+0000 7f3c3bd45640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.164 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:45.162+0000 7f9b12964640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.206 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:45.205+0000 7f26c13ef640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:45.708 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:45.707+0000 7f5878d06640 -1 osd.9 0 waiting for initial osdmap 2024-09-13T13:49:45.708 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:45.708+0000 7f586c719640 -1 osd.9 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:45.719 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:45.718+0000 7f5870b25640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.185 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:46.184+0000 7faeb8801640 -1 osd.8 0 waiting for initial osdmap 2024-09-13T13:49:46.186 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:46.185+0000 7faeac214640 -1 osd.8 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.206 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:46.205+0000 7ffafd2da640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T13:49:46.211 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:46.210+0000 7faeb0620640 -1 osd.8 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.502 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:46.500+0000 7ffaf0cec640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.511 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:46.509+0000 7ffaf50f9640 -1 osd.0 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.576 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:49:46.575+0000 7fe866541640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T13:49:46.576 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:46.575+0000 7fc178772640 -1 osd.11 0 waiting for initial osdmap 2024-09-13T13:49:46.577 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:49:46.575+0000 7fe859f53640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.579 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:46.579+0000 7fc16c184640 -1 osd.11 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.587 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:49:46.586+0000 7fe85e360640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.595 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:46.595+0000 7f003c1cd640 -1 osd.15 0 waiting for initial osdmap 2024-09-13T13:49:46.596 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:46.595+0000 7f002fbdf640 -1 osd.15 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.598 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:46.597+0000 7fc170591640 -1 osd.11 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.611 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:46.610+0000 7f0033fec640 -1 osd.15 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.685 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:49:46.685+0000 7f500ab80640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T13:49:46.686 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:49:46.685+0000 7f4ffdd7e640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.706 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:49:46.706+0000 7f500218b640 -1 osd.1 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.711 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:49:46.710+0000 7f7c80a23640 -1 osd.14 0 waiting for initial osdmap 2024-09-13T13:49:46.720 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:49:46.719+0000 7f7c7802e640 -1 osd.14 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:46.727 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:49:46.725+0000 7f1f44566640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T13:49:46.727 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:49:46.726+0000 7f1f37764640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-09-13T13:49:46.739 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:49:46.738+0000 7f1f3bb71640 -1 osd.4 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:47.167 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:49:47.165+0000 7faeac214640 -1 osd.8 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:49:47.209 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:47.208+0000 7ffaf0cec640 -1 osd.0 22 handle_osd_map injecting map crc failure 2024-09-13T13:49:47.212 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:49:47.211+0000 7f002fbdf640 -1 osd.15 22 handle_osd_map injecting map crc failure 2024-09-13T13:49:48.021 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T13:49:48.332 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-13T13:49:48.332 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":24,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:49:48.208768+0000","last_up_change":"2024-09-13T13:49:47.205190+0000","last_in_change":"2024-09-13T13:49:38.189589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":277461260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":277461260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":277461260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":277461260}]},"public_addr":"172.21.15.27:6812/277461260","cluster_addr":"172.21.15.27:6813/277461260","heartbeat_back_addr":"172.21.15.27:6815/277461260","heartbeat_front_addr":"172.21.15.27:6814/277461260","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.512922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.735970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.480014+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.954058+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.726760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.911654+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.913247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.924694+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.492471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.816808+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.278234+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:43.953059+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:45.050800+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.586118+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:49:48.344 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-13T13:49:48.344 INFO:tasks.ceph:Creating RBD pool 2024-09-13T13:49:48.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T13:49:49.217 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2024-09-13T13:49:49.227 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:49:49.226+0000 7f26bafde640 -1 osd.3 22 handle_osd_map injecting map crc failure 2024-09-13T13:49:49.235 DEBUG:teuthology.orchestra.run.smithi027:> rbd --cluster ceph pool init rbd 2024-09-13T13:49:49.592 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:49.592+0000 7f1a506ea640 -1 osd.13 22 handle_osd_map injecting map crc failure 2024-09-13T13:49:50.220 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:49:50.219+0000 7f7c73c21640 -1 osd.14 25 handle_osd_map injecting map crc failure 2024-09-13T13:49:50.362 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:49:50.361+0000 7fc16a180640 -1 osd.11 25 handle_osd_map injecting map crc failure 2024-09-13T13:49:51.428 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:51.426+0000 7ffaeece8640 -1 osd.0 26 handle_osd_map injecting map crc failure 2024-09-13T13:49:51.428 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:49:51.426+0000 7f9b0c553640 -1 osd.12 26 handle_osd_map injecting map crc failure 2024-09-13T13:49:52.242 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:49:52.242+0000 7f586a715640 -1 osd.9 27 handle_osd_map injecting map crc failure 2024-09-13T13:49:52.248 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T13:49:52.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T13:49:52.249 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T13:49:52.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:52.574 INFO:teuthology.orchestra.run.smithi027.stdout:[{"version":11,"timestamp":"2024-09-13T13:49:46.727040+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"10532.189906"}]}] 2024-09-13T13:49:52.575 INFO:tasks.ceph_manager:config epoch is 11 2024-09-13T13:49:52.575 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T13:49:52.575 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T13:49:52.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T13:49:52.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:52.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":3607579116}]},"active_addr":"172.21.15.159:6800/3607579116","active_change":"2024-09-13T13:49:38.355494+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1003459868}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":422396953}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3913981165}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":3829650726}]}]} 2024-09-13T13:49:52.951 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T13:49:52.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T13:49:52.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:49:53.258 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:53.258 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:49:52.235090+0000","last_up_change":"2024-09-13T13:49:47.205190+0000","last_in_change":"2024-09-13T13:49:38.189589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9499998092651367,"score_stable":5.9499998092651367,"optimal_score":0.62999999523162842,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":277461260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":277461260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":277461260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":277461260}]},"public_addr":"172.21.15.27:6812/277461260","cluster_addr":"172.21.15.27:6813/277461260","heartbeat_back_addr":"172.21.15.27:6815/277461260","heartbeat_front_addr":"172.21.15.27:6814/277461260","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.512922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.735970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.480014+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.954058+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.726760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.911654+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.913247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.205344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.924694+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.492471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.816808+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.278234+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:43.953059+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:45.050800+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.586118+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:49:53.271 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T13:49:53.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:49:53.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:53.579 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:49:52.235090+0000","last_up_change":"2024-09-13T13:49:47.205190+0000","last_in_change":"2024-09-13T13:49:38.189589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9499998092651367,"score_stable":5.9499998092651367,"optimal_score":0.62999999523162842,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":277461260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":277461260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":277461260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":277461260}]},"public_addr":"172.21.15.27:6812/277461260","cluster_addr":"172.21.15.27:6813/277461260","heartbeat_back_addr":"172.21.15.27:6815/277461260","heartbeat_front_addr":"172.21.15.27:6814/277461260","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.512922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.735970+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.480014+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.954058+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.726760+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.911654+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.913247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.205344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.924694+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.492471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.816808+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.278234+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:43.953059+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:45.050800+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.586118+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:49:53.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T13:49:53.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T13:49:53.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T13:49:53.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T13:49:53.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T13:49:53.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T13:49:53.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T13:49:53.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T13:49:53.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T13:49:53.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T13:49:53.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T13:49:53.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T13:49:53.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T13:49:53.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T13:49:53.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T13:49:53.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T13:49:54.028 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T13:49:54.048 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:54.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T13:49:54.059 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:54.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T13:49:54.068 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:54.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T13:49:54.070 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T13:49:54.080 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:54.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T13:49:54.081 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T13:49:54.084 INFO:teuthology.orchestra.run.smithi027.stdout:98784247812 2024-09-13T13:49:54.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T13:49:54.088 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T13:49:54.119 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:54.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T13:49:54.125 INFO:teuthology.orchestra.run.smithi027.stdout:90194313220 2024-09-13T13:49:54.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T13:49:54.134 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T13:49:54.163 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T13:49:54.165 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:54.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T13:49:54.170 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:54.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T13:49:54.176 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:54.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T13:49:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:49:54.781+0000 7f3e29fe1640 0 --2- 172.21.15.27:0/3702454139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e1c04ddb0 0x7f3e1c04e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T13:49:55.088 INFO:teuthology.orchestra.run.smithi027.stdout:94489280514 2024-09-13T13:49:55.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.2 2024-09-13T13:49:55.143 INFO:teuthology.orchestra.run.smithi027.stdout:98784247812 2024-09-13T13:49:55.167 INFO:teuthology.orchestra.run.smithi027.stdout:94489280514 2024-09-13T13:49:55.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247812 for osd.7 2024-09-13T13:49:55.184 INFO:teuthology.orchestra.run.smithi027.stdout:94489280514 2024-09-13T13:49:55.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.12 2024-09-13T13:49:55.199 INFO:teuthology.orchestra.run.smithi027.stdout:94489280514 2024-09-13T13:49:55.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.10 2024-09-13T13:49:55.228 INFO:teuthology.orchestra.run.smithi027.stdout:98784247811 2024-09-13T13:49:55.229 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.5 2024-09-13T13:49:55.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.1 2024-09-13T13:49:55.313 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:55.327 INFO:teuthology.orchestra.run.smithi027.stdout:98784247812 2024-09-13T13:49:55.328 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:55.328 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:55.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.11 2024-09-13T13:49:55.350 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:55.351 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.0 2024-09-13T13:49:55.359 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.361 INFO:teuthology.orchestra.run.smithi027.stdout:90194313220 2024-09-13T13:49:55.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.14 2024-09-13T13:49:55.362 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-13T13:49:55.365 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.4 2024-09-13T13:49:55.374 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.375 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:55.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.13 2024-09-13T13:49:55.381 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.384 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:55.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.15 2024-09-13T13:49:55.397 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.6 2024-09-13T13:49:55.399 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.417 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:55.421 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:55.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.8 2024-09-13T13:49:55.431 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:55.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.3 2024-09-13T13:49:55.434 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T13:49:56.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T13:49:56.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T13:49:56.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T13:49:56.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T13:49:56.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T13:49:56.548 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:56.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.2 2024-09-13T13:49:56.569 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.581 INFO:teuthology.orchestra.run.smithi027.stdout:98784247814 2024-09-13T13:49:56.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.7 2024-09-13T13:49:56.595 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.622 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:56.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-13T13:49:56.640 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.687 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:56.690 INFO:teuthology.orchestra.run.smithi027.stdout:94489280515 2024-09-13T13:49:56.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.5 2024-09-13T13:49:56.705 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.707 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.12 2024-09-13T13:49:56.708 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.734 INFO:teuthology.orchestra.run.smithi027.stdout:98784247813 2024-09-13T13:49:56.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.1 2024-09-13T13:49:56.747 DEBUG:teuthology.parallel:result is None 2024-09-13T13:49:56.747 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:49:56.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:49:57.087 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:57.088 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:49:57.099 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T13:49:56.362638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437012,"kb_used_data":4628,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512428,"statfs":{"total":1546188226560,"available":1545740726272,"internally_reserved":0,"allocated":4739072,"data_stored":2689939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":442473553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":11,"num_write":23,"num_write_kb":521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.130729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242335+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:49:52.242335+0000","last_peered":"2024-09-13T13:49:52.242335+0000","last_clean":"2024-09-13T13:49:52.242335+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:49:52.242335+0000","last_undegraded":"2024-09-13T13:49:52.242335+0000","last_fullsized":"2024-09-13T13:49:52.242335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.281318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.247857+0000","last_change":"2024-09-13T13:49:52.248018+0000","last_active":"2024-09-13T13:49:52.247857+0000","last_peered":"2024-09-13T13:49:52.247857+0000","last_clean":"2024-09-13T13:49:52.247857+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:49:52.247857+0000","last_undegraded":"2024-09-13T13:49:52.247857+0000","last_fullsized":"2024-09-13T13:49:52.247857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:49.267208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.239921+0000","last_change":"2024-09-13T13:49:52.239921+0000","last_active":"2024-09-13T13:49:52.239921+0000","last_peered":"2024-09-13T13:49:52.239921+0000","last_clean":"2024-09-13T13:49:52.239921+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:49:52.239921+0000","last_undegraded":"2024-09-13T13:49:52.239921+0000","last_fullsized":"2024-09-13T13:49:52.239921+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:52.937080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.370546+0000","last_change":"2024-09-13T13:49:52.370546+0000","last_active":"2024-09-13T13:49:52.370546+0000","last_peered":"2024-09-13T13:49:52.370546+0000","last_clean":"2024-09-13T13:49:52.370546+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:49:52.370546+0000","last_undegraded":"2024-09-13T13:49:52.370546+0000","last_fullsized":"2024-09-13T13:49:52.370546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.318833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.243546+0000","last_change":"2024-09-13T13:49:52.242997+0000","last_active":"2024-09-13T13:49:52.243546+0000","last_peered":"2024-09-13T13:49:52.243546+0000","last_clean":"2024-09-13T13:49:52.243546+0000","last_became_active":"2024-09-13T13:49:50.227674+0000","last_became_peered":"2024-09-13T13:49:50.227674+0000","last_unstale":"2024-09-13T13:49:52.243546+0000","last_undegraded":"2024-09-13T13:49:52.243546+0000","last_fullsized":"2024-09-13T13:49:52.243546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:59.569557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080454,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369120+0000","last_change":"2024-09-13T13:49:52.369120+0000","last_active":"2024-09-13T13:49:52.369120+0000","last_peered":"2024-09-13T13:49:52.369120+0000","last_clean":"2024-09-13T13:49:52.369120+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:49:52.369120+0000","last_undegraded":"2024-09-13T13:49:52.369120+0000","last_fullsized":"2024-09-13T13:49:52.369120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.751922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242339+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:49:52.242339+0000","last_peered":"2024-09-13T13:49:52.242339+0000","last_clean":"2024-09-13T13:49:52.242339+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:49:52.242339+0000","last_undegraded":"2024-09-13T13:49:52.242339+0000","last_fullsized":"2024-09-13T13:49:52.242339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:14.783466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369080+0000","last_change":"2024-09-13T13:49:52.369080+0000","last_active":"2024-09-13T13:49:52.369080+0000","last_peered":"2024-09-13T13:49:52.369080+0000","last_clean":"2024-09-13T13:49:52.369080+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:49:52.369080+0000","last_undegraded":"2024-09-13T13:49:52.369080+0000","last_fullsized":"2024-09-13T13:49:52.369080+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:13.538100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.380205+0000","last_change":"2024-09-13T13:49:48.539255+0000","last_active":"2024-09-13T13:49:52.380205+0000","last_peered":"2024-09-13T13:49:52.380205+0000","last_clean":"2024-09-13T13:49:52.380205+0000","last_became_active":"2024-09-13T13:49:48.538219+0000","last_became_peered":"2024-09-13T13:49:48.538219+0000","last_unstale":"2024-09-13T13:49:52.380205+0000","last_undegraded":"2024-09-13T13:49:52.380205+0000","last_fullsized":"2024-09-13T13:49:52.380205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:49:47.205190+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.389193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":680921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":93030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":95684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":94866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":794624,"data_stored":684674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":96515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":91937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,5,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":92884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":95075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":88291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":95706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":98218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":90314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":88524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:49:57.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:49:57.401 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:57.401 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:49:57.413 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T13:49:56.362638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437012,"kb_used_data":4628,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512428,"statfs":{"total":1546188226560,"available":1545740726272,"internally_reserved":0,"allocated":4739072,"data_stored":2689939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":442473553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":11,"num_write":23,"num_write_kb":521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.130729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242335+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:49:52.242335+0000","last_peered":"2024-09-13T13:49:52.242335+0000","last_clean":"2024-09-13T13:49:52.242335+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:49:52.242335+0000","last_undegraded":"2024-09-13T13:49:52.242335+0000","last_fullsized":"2024-09-13T13:49:52.242335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.281318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.247857+0000","last_change":"2024-09-13T13:49:52.248018+0000","last_active":"2024-09-13T13:49:52.247857+0000","last_peered":"2024-09-13T13:49:52.247857+0000","last_clean":"2024-09-13T13:49:52.247857+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:49:52.247857+0000","last_undegraded":"2024-09-13T13:49:52.247857+0000","last_fullsized":"2024-09-13T13:49:52.247857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:49.267208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.239921+0000","last_change":"2024-09-13T13:49:52.239921+0000","last_active":"2024-09-13T13:49:52.239921+0000","last_peered":"2024-09-13T13:49:52.239921+0000","last_clean":"2024-09-13T13:49:52.239921+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:49:52.239921+0000","last_undegraded":"2024-09-13T13:49:52.239921+0000","last_fullsized":"2024-09-13T13:49:52.239921+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:52.937080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.370546+0000","last_change":"2024-09-13T13:49:52.370546+0000","last_active":"2024-09-13T13:49:52.370546+0000","last_peered":"2024-09-13T13:49:52.370546+0000","last_clean":"2024-09-13T13:49:52.370546+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:49:52.370546+0000","last_undegraded":"2024-09-13T13:49:52.370546+0000","last_fullsized":"2024-09-13T13:49:52.370546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.318833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.243546+0000","last_change":"2024-09-13T13:49:52.242997+0000","last_active":"2024-09-13T13:49:52.243546+0000","last_peered":"2024-09-13T13:49:52.243546+0000","last_clean":"2024-09-13T13:49:52.243546+0000","last_became_active":"2024-09-13T13:49:50.227674+0000","last_became_peered":"2024-09-13T13:49:50.227674+0000","last_unstale":"2024-09-13T13:49:52.243546+0000","last_undegraded":"2024-09-13T13:49:52.243546+0000","last_fullsized":"2024-09-13T13:49:52.243546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:59.569557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080454,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369120+0000","last_change":"2024-09-13T13:49:52.369120+0000","last_active":"2024-09-13T13:49:52.369120+0000","last_peered":"2024-09-13T13:49:52.369120+0000","last_clean":"2024-09-13T13:49:52.369120+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:49:52.369120+0000","last_undegraded":"2024-09-13T13:49:52.369120+0000","last_fullsized":"2024-09-13T13:49:52.369120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.751922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242339+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:49:52.242339+0000","last_peered":"2024-09-13T13:49:52.242339+0000","last_clean":"2024-09-13T13:49:52.242339+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:49:52.242339+0000","last_undegraded":"2024-09-13T13:49:52.242339+0000","last_fullsized":"2024-09-13T13:49:52.242339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:14.783466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369080+0000","last_change":"2024-09-13T13:49:52.369080+0000","last_active":"2024-09-13T13:49:52.369080+0000","last_peered":"2024-09-13T13:49:52.369080+0000","last_clean":"2024-09-13T13:49:52.369080+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:49:52.369080+0000","last_undegraded":"2024-09-13T13:49:52.369080+0000","last_fullsized":"2024-09-13T13:49:52.369080+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:13.538100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.380205+0000","last_change":"2024-09-13T13:49:48.539255+0000","last_active":"2024-09-13T13:49:52.380205+0000","last_peered":"2024-09-13T13:49:52.380205+0000","last_clean":"2024-09-13T13:49:52.380205+0000","last_became_active":"2024-09-13T13:49:48.538219+0000","last_became_peered":"2024-09-13T13:49:48.538219+0000","last_unstale":"2024-09-13T13:49:52.380205+0000","last_undegraded":"2024-09-13T13:49:52.380205+0000","last_fullsized":"2024-09-13T13:49:52.380205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:49:47.205190+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.389193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":680921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":93030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":95684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":94866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":794624,"data_stored":684674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":96515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":91937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,5,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":92884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":95075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":88291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":95706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":98218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":90314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":88524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:49:57.415 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:49:57.415 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T13:49:57.415 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T13:49:57.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T13:49:57.757 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:49:57.757 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T13:49:57.770 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T13:49:57.770 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T13:49:57.780 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T13:49:57.780 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:49:57.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/balancer/log_level debug' 2024-09-13T13:49:58.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-09-13T13:49:58.409 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:49:58.409 INFO:teuthology.orchestra.run.smithi027.stdout: "active": true, 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_duration": "0:00:00.000476", 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_started": "Fri Sep 13 13:49:38 2024", 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "upmap", 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "no_optimization_needed": false, 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "optimize_result": "No pools available", 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout: "plans": [] 2024-09-13T13:49:58.410 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:49:58.422 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client reef' 2024-09-13T13:49:59.629 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:59.628+0000 7ffaeece8640 -1 osd.0 28 handle_osd_map injecting map crc failure 2024-09-13T13:49:59.629 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:49:59.627+0000 7f1a506ea640 -1 osd.13 28 handle_osd_map injecting map crc failure 2024-09-13T13:49:59.719 INFO:teuthology.orchestra.run.smithi027.stderr:set require_min_compat_client to reef 2024-09-13T13:49:59.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode upmap-read' 2024-09-13T13:49:59.859 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:49:59.858+0000 7ffaf0cec640 -1 osd.0 29 handle_osd_map injecting map crc failure 2024-09-13T13:50:00.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-09-13T13:50:00.094 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:00.092+0000 7f002dbdb640 -1 osd.15 29 handle_osd_map injecting map crc failure 2024-09-13T13:50:00.146 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:00.146+0000 7faeaa210640 -1 osd.8 28 handle_osd_map injecting map crc failure 2024-09-13T13:50:00.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-09-13T13:50:00.680 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:00.680 INFO:teuthology.orchestra.run.smithi027.stdout: "active": true, 2024-09-13T13:50:00.680 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_duration": "0:00:00.000936", 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_started": "Fri Sep 13 13:50:00 2024", 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "upmap-read", 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: "no_optimization_needed": true, 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: "optimize_result": "Optimization plan created successfully", 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout: "plans": [] 2024-09-13T13:50:00.681 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:00.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status detail' 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout: "active": true, 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_duration": "0:00:00.000936", 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout: "last_optimize_started": "Fri Sep 13 13:50:00 2024", 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout: "mode": "upmap-read", 2024-09-13T13:50:00.994 INFO:teuthology.orchestra.run.smithi027.stdout: "no_optimization_needed": true, 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "optimize_result": "Optimization plan created successfully", 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items_added": [], 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items_removed": [], 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries_added": [], 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries_removed": [], 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout: "plans": [] 2024-09-13T13:50:00.995 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:01.007 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T13:50:01.018 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T13:50:01.018 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T13:50:01.018 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-13T13:50:01.018 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T13:50:01.343 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T13:50:01.343 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:50:01.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:50:01.645 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:01.645 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:50:01.657 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T13:50:01.056188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437160,"kb_used_data":4776,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512280,"statfs":{"total":1546188226560,"available":1545740574720,"internally_reserved":0,"allocated":4890624,"data_stored":2779307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":442473553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.694104"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242335+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:49:52.242335+0000","last_peered":"2024-09-13T13:49:52.242335+0000","last_clean":"2024-09-13T13:49:52.242335+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:49:52.242335+0000","last_undegraded":"2024-09-13T13:49:52.242335+0000","last_fullsized":"2024-09-13T13:49:52.242335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.281318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.247857+0000","last_change":"2024-09-13T13:49:52.248018+0000","last_active":"2024-09-13T13:49:52.247857+0000","last_peered":"2024-09-13T13:49:52.247857+0000","last_clean":"2024-09-13T13:49:52.247857+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:49:52.247857+0000","last_undegraded":"2024-09-13T13:49:52.247857+0000","last_fullsized":"2024-09-13T13:49:52.247857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:49.267208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.239921+0000","last_change":"2024-09-13T13:49:52.239921+0000","last_active":"2024-09-13T13:49:52.239921+0000","last_peered":"2024-09-13T13:49:52.239921+0000","last_clean":"2024-09-13T13:49:52.239921+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:49:52.239921+0000","last_undegraded":"2024-09-13T13:49:52.239921+0000","last_fullsized":"2024-09-13T13:49:52.239921+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:52.937080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.370546+0000","last_change":"2024-09-13T13:49:52.370546+0000","last_active":"2024-09-13T13:49:52.370546+0000","last_peered":"2024-09-13T13:49:52.370546+0000","last_clean":"2024-09-13T13:49:52.370546+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:49:52.370546+0000","last_undegraded":"2024-09-13T13:49:52.370546+0000","last_fullsized":"2024-09-13T13:49:52.370546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.318833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.243546+0000","last_change":"2024-09-13T13:49:52.242997+0000","last_active":"2024-09-13T13:49:52.243546+0000","last_peered":"2024-09-13T13:49:52.243546+0000","last_clean":"2024-09-13T13:49:52.243546+0000","last_became_active":"2024-09-13T13:49:50.227674+0000","last_became_peered":"2024-09-13T13:49:50.227674+0000","last_unstale":"2024-09-13T13:49:52.243546+0000","last_undegraded":"2024-09-13T13:49:52.243546+0000","last_fullsized":"2024-09-13T13:49:52.243546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:59.569557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080454,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369120+0000","last_change":"2024-09-13T13:49:52.369120+0000","last_active":"2024-09-13T13:49:52.369120+0000","last_peered":"2024-09-13T13:49:52.369120+0000","last_clean":"2024-09-13T13:49:52.369120+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:49:52.369120+0000","last_undegraded":"2024-09-13T13:49:52.369120+0000","last_fullsized":"2024-09-13T13:49:52.369120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.751922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242339+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:49:52.242339+0000","last_peered":"2024-09-13T13:49:52.242339+0000","last_clean":"2024-09-13T13:49:52.242339+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:49:52.242339+0000","last_undegraded":"2024-09-13T13:49:52.242339+0000","last_fullsized":"2024-09-13T13:49:52.242339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:14.783466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369080+0000","last_change":"2024-09-13T13:49:52.369080+0000","last_active":"2024-09-13T13:49:52.369080+0000","last_peered":"2024-09-13T13:49:52.369080+0000","last_clean":"2024-09-13T13:49:52.369080+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:49:52.369080+0000","last_undegraded":"2024-09-13T13:49:52.369080+0000","last_fullsized":"2024-09-13T13:49:52.369080+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:13.538100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-13T13:50:00.148682+0000","last_change":"2024-09-13T13:49:48.539255+0000","last_active":"2024-09-13T13:50:00.148682+0000","last_peered":"2024-09-13T13:50:00.148682+0000","last_clean":"2024-09-13T13:50:00.148682+0000","last_became_active":"2024-09-13T13:49:48.538219+0000","last_became_peered":"2024-09-13T13:49:48.538219+0000","last_unstale":"2024-09-13T13:50:00.148682+0000","last_undegraded":"2024-09-13T13:50:00.148682+0000","last_fullsized":"2024-09-13T13:50:00.148682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:49:47.205190+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.389193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":680921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":93030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":109508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":108690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":115474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":691614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":96515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":105761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,5,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":106708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":108383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":88291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":95706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":98218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":90314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":88524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:50:01.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:50:01.962 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:01.963 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:50:01.975 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T13:50:01.056188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437160,"kb_used_data":4776,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512280,"statfs":{"total":1546188226560,"available":1545740574720,"internally_reserved":0,"allocated":4890624,"data_stored":2779307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":442473553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.694104"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242335+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:49:52.242335+0000","last_peered":"2024-09-13T13:49:52.242335+0000","last_clean":"2024-09-13T13:49:52.242335+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:49:52.242335+0000","last_undegraded":"2024-09-13T13:49:52.242335+0000","last_fullsized":"2024-09-13T13:49:52.242335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.281318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.247857+0000","last_change":"2024-09-13T13:49:52.248018+0000","last_active":"2024-09-13T13:49:52.247857+0000","last_peered":"2024-09-13T13:49:52.247857+0000","last_clean":"2024-09-13T13:49:52.247857+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:49:52.247857+0000","last_undegraded":"2024-09-13T13:49:52.247857+0000","last_fullsized":"2024-09-13T13:49:52.247857+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:49.267208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.239921+0000","last_change":"2024-09-13T13:49:52.239921+0000","last_active":"2024-09-13T13:49:52.239921+0000","last_peered":"2024-09-13T13:49:52.239921+0000","last_clean":"2024-09-13T13:49:52.239921+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:49:52.239921+0000","last_undegraded":"2024-09-13T13:49:52.239921+0000","last_fullsized":"2024-09-13T13:49:52.239921+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:52.937080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.370546+0000","last_change":"2024-09-13T13:49:52.370546+0000","last_active":"2024-09-13T13:49:52.370546+0000","last_peered":"2024-09-13T13:49:52.370546+0000","last_clean":"2024-09-13T13:49:52.370546+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:49:52.370546+0000","last_undegraded":"2024-09-13T13:49:52.370546+0000","last_fullsized":"2024-09-13T13:49:52.370546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.318833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.243546+0000","last_change":"2024-09-13T13:49:52.242997+0000","last_active":"2024-09-13T13:49:52.243546+0000","last_peered":"2024-09-13T13:49:52.243546+0000","last_clean":"2024-09-13T13:49:52.243546+0000","last_became_active":"2024-09-13T13:49:50.227674+0000","last_became_peered":"2024-09-13T13:49:50.227674+0000","last_unstale":"2024-09-13T13:49:52.243546+0000","last_undegraded":"2024-09-13T13:49:52.243546+0000","last_fullsized":"2024-09-13T13:49:52.243546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:59.569557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001080454,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369120+0000","last_change":"2024-09-13T13:49:52.369120+0000","last_active":"2024-09-13T13:49:52.369120+0000","last_peered":"2024-09-13T13:49:52.369120+0000","last_clean":"2024-09-13T13:49:52.369120+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:49:52.369120+0000","last_undegraded":"2024-09-13T13:49:52.369120+0000","last_fullsized":"2024-09-13T13:49:52.369120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.751922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.242339+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:49:52.242339+0000","last_peered":"2024-09-13T13:49:52.242339+0000","last_clean":"2024-09-13T13:49:52.242339+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:49:52.242339+0000","last_undegraded":"2024-09-13T13:49:52.242339+0000","last_fullsized":"2024-09-13T13:49:52.242339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:14.783466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T13:49:52.369080+0000","last_change":"2024-09-13T13:49:52.369080+0000","last_active":"2024-09-13T13:49:52.369080+0000","last_peered":"2024-09-13T13:49:52.369080+0000","last_clean":"2024-09-13T13:49:52.369080+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:49:52.369080+0000","last_undegraded":"2024-09-13T13:49:52.369080+0000","last_fullsized":"2024-09-13T13:49:52.369080+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:13.538100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":159,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-13T13:50:00.148682+0000","last_change":"2024-09-13T13:49:48.539255+0000","last_active":"2024-09-13T13:50:00.148682+0000","last_peered":"2024-09-13T13:50:00.148682+0000","last_clean":"2024-09-13T13:50:00.148682+0000","last_became_active":"2024-09-13T13:49:48.538219+0000","last_became_peered":"2024-09-13T13:49:48.538219+0000","last_unstale":"2024-09-13T13:50:00.148682+0000","last_undegraded":"2024-09-13T13:50:00.148682+0000","last_fullsized":"2024-09-13T13:50:00.148682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:49:47.205190+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.389193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":680921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":101650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":93030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":109508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":108690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":115474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":691614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":96515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":105761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,5,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":106708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":108383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":88291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":95706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":98218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":188416,"data_stored":90314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":88524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:50:01.976 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:50:01.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-13T13:50:02.281 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 32 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:fsid cd1b3cf6-d610-4c64-94c3-504da850dd35 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-09-13T13:49:33.733123+0000 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-09-13T13:50:02.055189+0000 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:crush_version 6 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:full_ratio 0.95 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:backfillfull_ratio 0.9 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:nearfull_ratio 0.85 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:require_min_compat_client reef 2024-09-13T13:50:02.282 INFO:teuthology.orchestra.run.smithi027.stdout:min_compat_client reef 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:require_osd_release squid 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:stretch_mode_enabled false 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 3.97 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:max_osd 16 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6808/4285669868 v2:172.21.15.27:6809/4285669868 exists,up 1259d7f4-8ac0-4fdc-8473-9bdafc036502 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6808/4081030369 v2:172.21.15.100:6809/4081030369 exists,up 9035b2e5-ff53-4675-83b8-bf76ab2d3735 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6800/2454012715 v2:172.21.15.122:6801/2454012715 exists,up 497bd315-4b6e-4f41-b0de-488a942ac4d9 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6801/3322176894 v2:172.21.15.159:6802/3322176894 exists,up 024edf52-c84f-4d52-b825-c203ed5aecb6 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6812/277461260 v2:172.21.15.27:6813/277461260 exists,up c440743d-dac9-4c75-b6b3-94360a48f75c 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6804/3923508322 v2:172.21.15.100:6805/3923508322 exists,up d8203cad-54c8-43f1-af73-ec199c8a41c9 2024-09-13T13:50:02.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6804/1372880522 v2:172.21.15.122:6805/1372880522 exists,up d5e9b934-6b1e-4f27-838c-6529f69a5b65 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6813/1499158947 v2:172.21.15.159:6814/1499158947 exists,up 46d21c0b-bd67-45c9-8bcc-9e76edd03769 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6804/330880331 v2:172.21.15.27:6805/330880331 exists,up bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9 up in weight 1 up_from 22 up_thru 31 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6812/1204111774 v2:172.21.15.100:6813/1204111774 exists,up ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6812/2574350402 v2:172.21.15.122:6813/2574350402 exists,up 417658a1-1270-435c-8e2a-b9e4a8b0dba6 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6809/1984230453 v2:172.21.15.159:6810/1984230453 exists,up 9191abab-883f-40a1-8176-3c2e74d90c07 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6800/2410035394 v2:172.21.15.27:6801/2410035394 exists,up b9a2390c-9646-489d-b46b-268aa3e73194 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6800/450451503 v2:172.21.15.100:6801/450451503 exists,up b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6808/1255553857 v2:172.21.15.122:6809/1255553857 exists,up 99ad3153-7823-4dd9-9c09-044eba14bcd5 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6805/4073091144 v2:172.21.15.159:6806/4073091144 exists,up 0880b9c1-ac74-4794-b193-f21d0f1a27f6 2024-09-13T13:50:02.284 INFO:teuthology.orchestra.run.smithi027.stdout:pg_upmap_primary 2.2 9 2024-09-13T13:50:02.295 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6808/4285669868 v2:172.21.15.27:6809/4285669868 exists,up 1259d7f4-8ac0-4fdc-8473-9bdafc036502', 'osd.1 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6808/4081030369 v2:172.21.15.100:6809/4081030369 exists,up 9035b2e5-ff53-4675-83b8-bf76ab2d3735', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6800/2454012715 v2:172.21.15.122:6801/2454012715 exists,up 497bd315-4b6e-4f41-b0de-488a942ac4d9', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6801/3322176894 v2:172.21.15.159:6802/3322176894 exists,up 024edf52-c84f-4d52-b825-c203ed5aecb6', 'osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6812/277461260 v2:172.21.15.27:6813/277461260 exists,up c440743d-dac9-4c75-b6b3-94360a48f75c', 'osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6804/3923508322 v2:172.21.15.100:6805/3923508322 exists,up d8203cad-54c8-43f1-af73-ec199c8a41c9', 'osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6804/1372880522 v2:172.21.15.122:6805/1372880522 exists,up d5e9b934-6b1e-4f27-838c-6529f69a5b65', 'osd.7 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6813/1499158947 v2:172.21.15.159:6814/1499158947 exists,up 46d21c0b-bd67-45c9-8bcc-9e76edd03769', 'osd.8 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6804/330880331 v2:172.21.15.27:6805/330880331 exists,up bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0', 'osd.9 up in weight 1 up_from 22 up_thru 31 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6812/1204111774 v2:172.21.15.100:6813/1204111774 exists,up ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6812/2574350402 v2:172.21.15.122:6813/2574350402 exists,up 417658a1-1270-435c-8e2a-b9e4a8b0dba6', 'osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6809/1984230453 v2:172.21.15.159:6810/1984230453 exists,up 9191abab-883f-40a1-8176-3c2e74d90c07', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.27:6800/2410035394 v2:172.21.15.27:6801/2410035394 exists,up b9a2390c-9646-489d-b46b-268aa3e73194', 'osd.13 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.100:6800/450451503 v2:172.21.15.100:6801/450451503 exists,up b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18', 'osd.14 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.122:6808/1255553857 v2:172.21.15.122:6809/1255553857 exists,up 99ad3153-7823-4dd9-9c09-044eba14bcd5', 'osd.15 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.159:6805/4073091144 v2:172.21.15.159:6806/4073091144 exists,up 0880b9c1-ac74-4794-b193-f21d0f1a27f6'] 2024-09-13T13:50:02.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-13T13:50:02.398 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:02.398 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "mon.a", 2024-09-13T13:50:02.398 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster": "ceph", 2024-09-13T13:50:02.398 INFO:teuthology.orchestra.run.smithi027.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T13:50:02.398 INFO:teuthology.orchestra.run.smithi027.stdout: "admin_socket_mode": "", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_client_required": "cephx, none", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_cluster_required": "cephx", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_debug": "true", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_service_required": "cephx", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_supported": "cephx", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio": "true", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T13:50:02.399 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_aio_reap_max": "16", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_async_discard_threads": "0", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_block_size": "4096", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio": "false", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_enable_discard": "false", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_flock_retry": "3", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_inject_crash": "0", 2024-09-13T13:50:02.400 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring": "false", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring_hipri": "false", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bdev_type": "", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_allocator": "hybrid", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_buffered_io": "true", 2024-09-13T13:50:02.401 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_replay_recovery": "false", 2024-09-13T13:50:02.402 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluefs_sync_write": "false", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_allocator": "hybrid", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T13:50:02.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_create": "true", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_create": "false", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_path": "", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_db_size": "0", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_path": "", 2024-09-13T13:50:02.404 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_size": "107374182400", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_create": "false", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_path": "", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs": "true", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T13:50:02.405 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_autotune": "true", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size": "0", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T13:50:02.406 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cache_type": "2q", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_clone_cow": "true", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_mode": "none", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T13:50:02.407 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_freelist": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T13:50:02.408 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T13:50:02.409 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fail_eio": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T13:50:02.410 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T13:50:02.411 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size": "0", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T13:50:02.412 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T13:50:02.413 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_spdk_mem": "512", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_tracing": "false", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T13:50:02.414 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_write_v2": "false", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_write_v2_random": "false", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T13:50:02.415 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_cluster_require_version": "2", 2024-09-13T13:50:02.416 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_require_signatures": "false", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_require_version": "2", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_service_require_signatures": "false", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_service_require_version": "2", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "cephx_sign_messages": "true", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "chdir": "", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_acl_type": "", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_asio_thread_count": "2", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_cache_mid": "0.750000", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_cache_size": "16384", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_caps_release_delay": "5", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_check_pool_perm": "true", 2024-09-13T13:50:02.417 INFO:teuthology.orchestra.run.smithi027.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_force_sync_read": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_getattr_caps": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_inject_features": "", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_die_on_failed_remount": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_dirsize_rbytes": "true", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_force_lazyio": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_fs": "", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_inject_release_failure": "false", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_max_inline_size": "4096", 2024-09-13T13:50:02.418 INFO:teuthology.orchestra.run.smithi027.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mds_namespace": "", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_metadata": "", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_gid": "-1", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_timeout": "300", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mount_uid": "-1", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_mountpoint": "/", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_notify_timeout": "10", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc": "true", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_max_objects": "1000", 2024-09-13T13:50:02.419 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_size": "209715200", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_permissions": "true", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_quota": "true", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_quota_df": "true", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_max_bytes": "0", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_max_periods": "4", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_readahead_min": "131072", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_reconnect_stale": "false", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_shutdown_timeout": "30", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_snapdir": ".snap", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_tick_interval": "1", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_trace": "", 2024-09-13T13:50:02.420 INFO:teuthology.orchestra.run.smithi027.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_use_faked_inos": "false", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "client_use_random_mds": "false", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog": "false", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_graylog_port": "12201", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_monitors": "default=true", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog": "false", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "clog_to_syslog_level": "info", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "-", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_network": "", 2024-09-13T13:50:02.421 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_network_interface": "", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_isal": "false", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_level": "5", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "compressor_zstd_level": "1", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T13:50:02.422 INFO:teuthology.orchestra.run.smithi027.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location": "", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location_hook": "", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_location_hook_timeout": "10", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "daemonize": "false", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "daos_pool": "tank", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_alienstore": "0/5", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asok": "1/5", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asok_assert_abort": "false", 2024-09-13T13:50:02.423 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_auth": "20/20", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bdev": "1/3", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bluefs": "1/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_bluestore": "1/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_buffer": "0/1", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cephsqlite": "0/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_client": "0/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_compressor": "1/5", 2024-09-13T13:50:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_context": "0/1", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crush": "1/1", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_crypto": "1/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_cyanstore": "0/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_dpdk": "1/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_eventtrace": "1/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_filer": "0/1", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_filestore": "1/3", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_finisher": "1/1", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_fuse": "1/5", 2024-09-13T13:50:02.425 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_javaclient": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_journal": "1/3", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_journaler": "0/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_kstore": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_lockdep": "0/1", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mclock": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_balancer": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_locker": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_log": "1/5", 2024-09-13T13:50:02.426 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_migrator": "1/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_memstore": "1/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mgr": "2/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mgrc": "1/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_mon": "20/20", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_monc": "0/10", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_ms": "1/1", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_none": "0/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objclass": "0/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objectcacher": "0/5", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_objecter": "0/1", 2024-09-13T13:50:02.427 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_optracker": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_osd": "1/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_paxos": "20/20", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_perfcounter": "1/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_prioritycache": "1/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rados": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rbd_replay": "0/5", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_refs": "0/0", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_reserver": "1/1", 2024-09-13T13:50:02.428 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_access": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_flight": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_notification": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rgw_sync": "1/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_rocksdb": "4/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore": "0/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_backref": "0/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_cache": "0/5", 2024-09-13T13:50:02.429 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_device": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_epm": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_journal": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_lba": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_odata": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_omap": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_onode": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_t": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_seastore_tm": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_striper": "0/1", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_test": "0/5", 2024-09-13T13:50:02.430 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_throttle": "1/1", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_timer": "0/1", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_tp": "0/5", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "debug_trace": "1/5", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "device_failure_prediction_mode": "none", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_graylog": "false", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_journald": "false", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_stderr": "true", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "err_to_syslog": "false", 2024-09-13T13:50:02.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event_tracing": "false", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_cert_file": "", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_http_port": "9926", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_key_file": "", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_prio_limit": "5", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_sort_metrics": "true", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "exporter_stats_period": "5", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "fake_statfs_for_testing": "0", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "fatal_signal_handlers": "true", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "filer_max_purge_ops": "10", 2024-09-13T13:50:02.432 INFO:teuthology.orchestra.run.smithi027.stdout: "filer_max_truncate_ops": "128", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_blackhole": "false", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_btrfs_snap": "true", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_caller_concurrency": "10", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_omap_check": "false", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_debug_verify_split": "false", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_dump_file": "", 2024-09-13T13:50:02.433 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fadvise": "true", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fail_eio": "true", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fd_cache_size": "128", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fiemap": "false", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_inject_stall": "0", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_parallel": "false", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_trailing": "false", 2024-09-13T13:50:02.434 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_journal_writeahead": "false", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_kill_at": "0", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T13:50:02.435 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_merge_threshold": "-10", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_odsync_write": "false", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_backend_path": "", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T13:50:02.436 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_op_threads": "2", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_punch_hole": "false", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T13:50:02.437 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_queue_max_ops": "50", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_seek_data_hole": "false", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_sloppy_crc": "false", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_splice": "false", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_split_multiple": "2", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_split_rand_factor": "20", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_update_to": "1000", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T13:50:02.438 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T13:50:02.439 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_xfs_extsize": "false", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "filestore_zfs_snap": "false", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fio_dir": "/tmp/fio", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "cd1b3cf6-d610-4c64-94c3-504da850dd35", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_allow_other": "true", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_big_writes": "true", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_debug": "false", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_default_permissions": "false", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_disable_pagecache": "false", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_max_write": "0", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_multithreaded": "true", 2024-09-13T13:50:02.440 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_require_active_mds": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_set_user_groups": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_move": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_read": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_splice_write": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "gss_target_name": "ceph", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_file": "", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_inject_failure": "0", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_interval": "5", 2024-09-13T13:50:02.441 INFO:teuthology.orchestra.run.smithi027.stdout: "host": "smithi027", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T13:50:02.442 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "inject_early_sigterm": "false", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "jaeger_agent_port": "6799", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "jaeger_tracing_enable": "false", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_aio": "true", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_align_min_size": "65536", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_block_align": "true", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_block_size": "4096", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_dio": "true", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_discard": "false", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_force_aio": "false", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_ignore_corruption": "false", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_max_write_entries": "100", 2024-09-13T13:50:02.443 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_replay_from": "0", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_write_header_frequency": "0", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journal_zero_on_create": "false", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_prefetch_periods": "10", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_prezero_periods": "5", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "journaler_write_head_interval": "15", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "keyfile": "", 2024-09-13T13:50:02.444 INFO:teuthology.orchestra.run.smithi027.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_backend": "rocksdb", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_max_bytes": "67108864", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_max_ops": "512", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_onode_map_size": "1024", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "kstore_sync_transaction": "false", 2024-09-13T13:50:02.445 INFO:teuthology.orchestra.run.smithi027.stdout: "librados_thread_count": "2", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "lockdep": "false", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "lockdep_force_backtrace": "false", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_coarse_timestamps": "true", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_flush_on_exit": "false", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_graylog_port": "12201", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_max_new": "1000", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_max_recent": "10000", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_stderr_prefix": "", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_file": "true", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_graylog": "false", 2024-09-13T13:50:02.446 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_journald": "false", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_stderr": "false", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "log_to_syslog": "false", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_action_on_write_error": "1", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_alternate_name_max": "8192", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_asio_thread_count": "2", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_export_pin": "true", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T13:50:02.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_interval": "10", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_max": "-1", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_max_until": "-1", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_merge_size": "50", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_mode": "0", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T13:50:02.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_bits": "3", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_size": "10000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_mid": "0.700000", 2024-09-13T13:50:02.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_interval": "1", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T13:50:02.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_auth_pins": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_frag": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_scatterstat": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_debug_subtrees": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_default_dir_hash": "2", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_defer_session_stale": "true", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T13:50:02.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_max_entries": "0", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dir_prefetch": "true", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_early_reply": "true", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_enable_op_tracker": "true", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_enforce_unique_name": "true", 2024-09-13T13:50:02.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T13:50:02.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_health_dummy": "false", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_join_fs": "", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_journal_format": "1", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_create_at": "0", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T13:50:02.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_export_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_import_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_link_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_openc_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_rename_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T13:50:02.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_max_events": "-1", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_max_segments": "128", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_pause": "false", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_segment_size": "0", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_threshold": "128", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_completed_requests": "100000", 2024-09-13T13:50:02.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_export_size": "20971520", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_file_recover": "32", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_files": "64", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_ops": "8192", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_metrics_update_interval": "2", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_min_caps_per_client": "100", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_numa_node": "-1", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T13:50:02.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_duration": "600", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_size": "20", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_op_log_threshold": "5", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_ping_grace": "15", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_ping_interval": "5", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_caps": "30000", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T13:50:02.458 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_replay_interval": "1.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_root_ino_gid": "0", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_root_ino_uid": "0", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T13:50:02.459 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_shutdown_check": "0", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_skip_ino": "0", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_min_uid": "0", 2024-09-13T13:50:02.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_snap_rstat": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_symlink_recovery": "true", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_thrash_exports": "0", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_thrash_fragments": "0", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_tick_interval": "5.000000", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_valgrind_exit": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_verify_backtrace": "1", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_verify_scatter": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_wipe_sessions": "false", 2024-09-13T13:50:02.461 INFO:teuthology.orchestra.run.smithi027.stdout: "mempool_debug": "false", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_page_set": "false", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "memstore_page_size": "65536", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_bytes": "134217728", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_messages": "512", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_disabled_modules": "", 2024-09-13T13:50:02.462 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mds_messages": "128", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_mon_messages": "128", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_history_duration": "600", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_history_size": "20", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T13:50:02.463 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_op_log_threshold": "5", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_osd_messages": "8192", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_pool": "true", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_standby_modules": "true", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_stats_period": "5", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_stats_threshold": "5", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_tick_period": "2", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T13:50:02.464 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_allow_pool_delete": "true", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_allow_pool_size_one": "true", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_bytes": "104857600", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_directed_command_retry": "5", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T13:50:02.465 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T13:50:02.466 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T13:50:02.466 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T13:50:02.466 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_client_target_rank": "-1", 2024-09-13T13:50:02.466 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_level": "debug", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T13:50:02.468 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_start": "false", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_compact_on_trim": "true", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T13:50:02.469 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_cpu_threads": "4", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_avail_crit": "5", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_avail_warn": "5", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_json": "false", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_extra_checks": "false", 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_down_mkfs_grace": "120", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_down_uptime_grace": "60", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_election_default_strategy": "1", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_election_timeout": "5.000000", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T13:50:02.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_enable_op_tracker": "true", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_fake_pool_delete": "false", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_force_quorum_join": "false", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_detail_to_clog": "false", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_log_update_period": "5", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_max_detail": "50", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog": "true", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T13:50:02.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_host": "172.21.15.27,172.21.15.100,172.21.15.122", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_host_override": "", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_initial_members": "", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease": "5.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_full_interval": "50", 2024-09-13T13:50:02.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_max": "10000", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_log_max_summary": "50", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_log_epochs": "500", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_osd": "10000", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T13:50:02.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_memory_autotune": "true", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_memory_target": "2147483648", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_digest_period": "5", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_complaint_time": "30", 2024-09-13T13:50:02.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_duration": "600", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_size": "20", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_op_log_threshold": "5", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T13:50:02.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_cache_size": "500", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T13:50:02.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T13:50:02.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_report_timeout": "900", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T13:50:02.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_max_osds": "4", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T13:50:02.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_interval": "300", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_max_keys": "100", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_scrub_timeout": "300", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_session_timeout": "300", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_smart_report_timeout": "5", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_pool_size": "4", 2024-09-13T13:50:02.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_debug": "false", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_tick_interval": "5", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T13:50:02.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_older_version": "true", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T13:50:02.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": "", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T13:50:02.484 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "motr_tracing_enabled": "false", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_op_threads": "3", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_cm": "false", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_device_name": "", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T13:50:02.485 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_sl": "3", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_rdma_type": "ib", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_async_reap_threshold": "5", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_before_connect": "false", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_ipv4": "true", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_ipv6": "false", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_msgr1": "false", 2024-09-13T13:50:02.486 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_msgr2": "true", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_port_max": "7568", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_port_min": "6800", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_retry_count": "3", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_bind_retry_delay": "5", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_client": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mds": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mgr": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_mon": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_blackhole_osd": "false", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_client_mode": "crc secure", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T13:50:02.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_cluster_type": "", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_compress_secure": "false", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_crc_data": "true", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_crc_header": "true", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_bug": "true", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_old_message": "true", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T13:50:02.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hugepages": "", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_lro": "true", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T13:50:02.489 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_pmd": "", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_port_id": "0", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_dump_on_send": "false", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_delay_type": "osd", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_network_congestion": "0", 2024-09-13T13:50:02.490 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_inject_socket_failures": "2500", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_max_accept_failures": "4", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_max_backoff": "15.000000", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compress_mode": "none", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_pq_min_cost": "65536", 2024-09-13T13:50:02.491 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_public_type": "", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_service_mode": "crc secure", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_nodelay": "true", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "ms_type": "async", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "no_config_file": "false", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T13:50:02.492 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inflight_ops": "1024", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objecter_timeout": "10.000000", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objectstore_blackhole": "false", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "openssl_engine_opts": "", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T13:50:02.493 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_max_ops": "4", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_agent_slop": "0.020000", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_asio_thread_count": "2", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_async_recovery_min_cost": "1", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_scan_max": "512", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backfill_scan_min": "64", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T13:50:02.494 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_peering": "false", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_beacon_report_interval": "300", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_duration": "30", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_blkin_trace_all": "false", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T13:50:02.495 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_class_update_on_start": "true", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_message_cap": "256", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_op_priority": "63", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_client_watch_timeout": "30", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_command_max_records": "256", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_compact_on_start": "false", 2024-09-13T13:50:02.496 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_update_on_start": "true", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T13:50:02.497 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_op_order": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_shutdown": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T13:50:02.498 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_trim_objects": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T13:50:02.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_default_notify_timeout": "30", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_ec_partial_reads": "true", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_enable_op_tracker": "true", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T13:50:02.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_info": "true", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown": "true", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_function_tracing": "false", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_grace": "20", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_interval": "6", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T13:50:02.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_stale": "600", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_journal_size": "5120", 2024-09-13T13:50:02.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_kill_backfill_at": "0", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_cache_size": "50", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_dedup": "true", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_message_max": "40", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_attr_name_len": "100", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_attr_size": "0", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_backfills": "1", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_markdown_count": "5", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_markdown_period": "600", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_name_len": "2048", 2024-09-13T13:50:02.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_object_size": "134217728", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_pgls": "1024", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_push_cost": "8388608", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_push_objects": "10", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_scrubs": "3", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T13:50:02.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_max_write_size": "90", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_profile": "balanced", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T13:50:02.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_base": "805306368", 2024-09-13T13:50:02.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target": "4294967296", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target_autotune": "false", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_min_recovery_priority": "0", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_report_interval": "5", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T13:50:02.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_num_cache_shards": "32", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_node": "-1", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objecter_finishers": "1", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore": "bluestore", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore_fuse": "false", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_objectstore_tracing": "false", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_duration": "600", 2024-09-13T13:50:02.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_size": "20", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_log_threshold": "5", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards": "0", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T13:50:02.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_op_thread_timeout": "15", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_open_classes_on_start": "true", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_os_flags": "0", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_peering_op_priority": "255", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_delete_priority": "5", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T13:50:02.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_crimson": "false", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T13:50:02.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_flags": "0", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_min_size": "2", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T13:50:02.512 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_size": "2", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_default_type": "replicated", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_cost": "20971520", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active": "0", 2024-09-13T13:50:02.513 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_op_priority": "3", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_priority": "5", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T13:50:02.514 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_repair_during_recovery": "false", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_cost": "52428800", 2024-09-13T13:50:02.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_end_hour": "0", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T13:50:02.516 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_priority": "5", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_skip_data_digest": "false", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_smart_report_timeout": "5", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_priority": "5", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T13:50:02.517 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_target_transaction_size": "30", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T13:50:02.518 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_tracing": "false", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_use_stale_snap": "false", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_kill_at": "0", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_max_join_drift": "10", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_min": "500", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_min_wait": "0.050000", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_max": "500", 2024-09-13T13:50:02.519 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_service_trim_min": "10", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_stash_full_interval": "25", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_trim_max": "500", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "paxos_trim_min": "250", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "perf": "true", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "-", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrv": "", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "public_bind_addr": "-", 2024-09-13T13:50:02.520 INFO:teuthology.orchestra.run.smithi027.stdout: "public_network": "", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "public_network_interface": "", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "qat_compressor_enabled": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_mon_op_timeout": "0", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_osd_op_timeout": "0", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rados_tracing": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_atime_update_interval": "60", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T13:50:02.521 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache": "true", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_policy": "writearound", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_size": "33554432", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_compression_hint": "none", 2024-09-13T13:50:02.522 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_clone_format": "auto", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_data_pool": "", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_features": "61", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_format": "2", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_map_options": "", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_order": "22", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_pool": "rbd", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_stripe_count": "0", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T13:50:02.523 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_io_scheduler": "simple", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T13:50:02.524 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_order": "24", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_pool": "", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_journal_splay_width": "4", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T13:50:02.525 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T13:50:02.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T13:50:02.527 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_op_threads": "1", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_plugins": "", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T13:50:02.528 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T13:50:02.529 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T13:50:02.530 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_tracing": "false", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_validate_names": "true", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd_validate_pool": "true", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "restapi_base_url": "", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "restapi_log_level": "", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_admin_entry": "admin", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_backend_store": "rados", 2024-09-13T13:50:02.531 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_barbican_url": "", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_beast_enable_async": "true", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_enabled": "true", 2024-09-13T13:50:02.532 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_config_store": "rados", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_content_length_compat": "false", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_cross_domain_policy": "", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T13:50:02.533 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T13:50:02.534 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T13:50:02.535 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T13:50:02.536 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_log_window": "30", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T13:50:02.538 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_disable_s3select": "false", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T13:50:02.539 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dns_name": "", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dns_s3website_name": "", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T13:50:02.540 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_ops_log": "false", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_static_website": "false", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enable_usage_log": "false", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T13:50:02.541 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_expose_bucket": "false", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_extended_http_attrs": "", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_filter": "none", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_objs": "32", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T13:50:02.542 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_init_timeout": "300", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T13:50:02.543 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_password": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_project": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_token": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_admin_user": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T13:50:02.544 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_url": "", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_objs": "32", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T13:50:02.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_worker": "3", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ldap_uri": "", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T13:50:02.546 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_http_headers": "", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attr_size": "0", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T13:50:02.547 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_listing_results": "1000", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_notify_retries": "10", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T13:50:02.548 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T13:50:02.549 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_num_control_oids": "8", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_numa_node": "-1", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T13:50:02.550 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_op_tracing": "false", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_token": "", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_url": "", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_rados": "false", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T13:50:02.551 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_print_continue": "true", 2024-09-13T13:50:02.552 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rados_tracing": "false", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm": "", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm_id": "", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_region": "", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T13:50:02.553 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_request_uri": "", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_resolve_cname": "false", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_run_sync_thread": "true", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T13:50:02.554 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_script_uri": "", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_service_provider_name": "", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_client_id": "", 2024-09-13T13:50:02.555 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_client_secret": "", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_entry": "sts", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_key": "sts", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_auth_url": "", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_custom_header": "", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_need_stats": "true", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_tenant_name": "", 2024-09-13T13:50:02.556 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_url": "", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_lease_period": "120", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T13:50:02.557 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_thread_pool_size": "512", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_comment": "", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_createby": "", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_encoding": "", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_flag": "false", 2024-09-13T13:50:02.558 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_origin": "", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_torrent_tracker": "", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_max_shards": "32", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_use_opa_authz": "false", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_counters_cache": "false", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T13:50:02.559 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_user_unique_email": "true", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_verify_ssl": "true", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone": "", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone_id": "", 2024-09-13T13:50:02.560 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup": "", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup_id": "", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_block_size": "4096", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T13:50:02.561 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T13:50:02.562 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_partition_filters": "false", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_perf": "false", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "run_dir": "/var/run/ceph", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_block_create": "true", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_device_size": "53687091200", 2024-09-13T13:50:02.563 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_full_integrity_check": "false", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_main_device_type": "SSD", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T13:50:02.564 INFO:teuthology.orchestra.run.smithi027.stdout: "seastore_segment_size": "67108864", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "set_keepcaps": "false", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "setgroup": "", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "setuser": "", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "setuser_match_path": "", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "thp": "false", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "threadpool_default_timeout": "60", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "throttler_perf_counter": "true", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tmp_dir": "/tmp", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "uadk_compressor_enabled": "false", 2024-09-13T13:50:02.565 INFO:teuthology.orchestra.run.smithi027.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T13:50:02.566 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:02.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T13:50:02.693 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a: {} 2024-09-13T13:50:02.694 INFO:teuthology.orchestra.run.smithi027.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:50:02.708 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b: {} 2024-09-13T13:50:02.708 INFO:teuthology.orchestra.run.smithi027.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:50:02.722 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c: {} 2024-09-13T13:50:02.722 INFO:teuthology.orchestra.run.smithi027.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:50:02.733 DEBUG:teuthology.orchestra.run.smithi027:> type ceph-objectstore-tool 2024-09-13T13:50:02.748 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T13:50:02.749 DEBUG:teuthology.orchestra.run.smithi159:> type ceph-objectstore-tool 2024-09-13T13:50:02.767 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T13:50:02.767 DEBUG:teuthology.orchestra.run.smithi122:> type ceph-objectstore-tool 2024-09-13T13:50:02.785 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T13:50:02.785 DEBUG:teuthology.orchestra.run.smithi100:> type ceph-objectstore-tool 2024-09-13T13:50:02.804 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T13:50:02.805 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T13:50:02.813 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T13:50:02.813 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:50:02.813 DEBUG:teuthology.orchestra.run.smithi159:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-09-13T13:50:02.814 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T13:50:02.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:02.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:02.814 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:02.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-13T13:50:02.815 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T13:50:02.815 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T13:50:02.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:02.815 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T13:50:02.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:02.816 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T13:50:02.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:50:02.827 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:02.826+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:02.917 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:02.916+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:02.935 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:02.935 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:02.935 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:02.936 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:02.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:02.959 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:02.960 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:02.997 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:03.018 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:03.016+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:03.019 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:03.020 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:03.041 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:03.041 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:03.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:03.062 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:03.069 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.069 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:03.084 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:03.084 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:03.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:03.107 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:03.118 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:03.116+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:03.145 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:03.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:03.185 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:03.185 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:03.193 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.193 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:03.211 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 18 (0%) 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:avg 1.125 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 0.927025 -> 0.927025 (expected baseline 1.02698) 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.2 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.8 with 3 -> 3 pgs (2.66667 -> 2.66667 * mean) 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 0.05 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0003 2024-09-13T13:50:03.212 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0003 2024-09-13T13:50:03.213 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:03.213 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:03.213 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:03.219 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:03.218+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:03.225 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:03.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-13T13:50:03.319 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:03.319+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:03.321 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.321 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:03.420 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:03.419+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:03.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:03.441 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:03.463 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:03.463 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:03.480 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.480 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.484 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:03.484 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:03.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:03.506 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:03.507 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:03.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:03.507+0000 7fee7affd640 0 -- 172.21.15.27:0/3051765341 >> v2:172.21.15.122:6808/1255553857 conn(0x7fee7c21fb20 msgr2=0x7fee7c221f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:03.522 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:03.521+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:03.580 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 18 (0%) 2024-09-13T13:50:03.580 INFO:teuthology.orchestra.run.smithi027.stdout:avg 1.125 2024-09-13T13:50:03.580 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 0.927025 -> 0.927025 (expected baseline 1.02698) 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.2 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.8 with 3 -> 3 pgs (2.66667 -> 2.66667 * mean) 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 3 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0003 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0003 2024-09-13T13:50:03.581 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:03.593 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T13:50:03.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 0.05 2024-09-13T13:50:03.600 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.601 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:03.623 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:03.622+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:03.724 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:03.723+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:03.733 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:03.755 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:03.755 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:03.759 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.759 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:03.824 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:03.823+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:03.876 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.876 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:03.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 18 (0%) 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:avg 1.125 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 0.927025 -> 0.927025 (expected baseline 1.02698) 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.2 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.8 with 3 -> 3 pgs (2.66667 -> 2.66667 * mean) 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:03.908 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-09-13T13:50:03.909 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 0.05 2024-09-13T13:50:03.909 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:03.909 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0003 2024-09-13T13:50:03.909 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0003 2024-09-13T13:50:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:03.921 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T13:50:03.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-13T13:50:03.925 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:03.924+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:03.993 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:03.994 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:04.026 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:04.025+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:04.059 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:50:04.060 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'low_tier' created 2024-09-13T13:50:04.077 INFO:teuthology.run_tasks:Running task rados... 2024-09-13T13:50:04.088 INFO:tasks.rados:Beginning rados... 2024-09-13T13:50:04.088 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-13T13:50:04.089 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T13:50:04.089 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-13T13:50:04.089 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-13T13:50:04.108 INFO:tasks.rados:joining rados 2024-09-13T13:50:04.109 INFO:tasks.rados:clients are ['client.0'] 2024-09-13T13:50:04.109 INFO:tasks.rados:starting run 0 out of 1 2024-09-13T13:50:04.109 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T13:50:04.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-13T13:50:04.127 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:04.126+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:04.156 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.156 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:04.228 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:04.226+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:04.276 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 26 (0%) 2024-09-13T13:50:04.276 INFO:teuthology.orchestra.run.smithi027.stdout:avg 1.625 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 1.11102 -> 1.11102 (expected baseline 1.23428) 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.2 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.8 with 4 -> 4 pgs (2.46154 -> 2.46154 * mean) 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 0.05 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0003 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0029 2024-09-13T13:50:04.277 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:04.290 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-13T13:50:04.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-09-13T13:50:04.316 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.316 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:04.329 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:04.327+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:04.430 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:04.429+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:04.490 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.491 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:04.544 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:04.543+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:04.617 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.617 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.618 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 26 (0%) 2024-09-13T13:50:04.618 INFO:teuthology.orchestra.run.smithi027.stdout:avg 1.625 2024-09-13T13:50:04.618 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 1.11102 -> 1.11102 (expected baseline 1.23428) 2024-09-13T13:50:04.618 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.2 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-13T13:50:04.618 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.8 with 4 -> 4 pgs (2.46154 -> 2.46154 * mean) 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 3 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0003 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0003 2024-09-13T13:50:04.619 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:04.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:04.631 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:04.631+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:04.733 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:04.732+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:04.772 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.773 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.781 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:04.833 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:04.833+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:04.883 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.883 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:04.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:04.935 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:04.934+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:04.995 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:04.995 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:05.036 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:05.035+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' created 2024-09-13T13:50:05.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T13:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:05.136 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:05.134+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:05.272 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:05.271+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:05.332 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.332 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.337 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:05.336+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:05.340 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:05.440 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:05.438+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:05.483 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.483 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:05.539 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:05.539+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:05.593 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.594 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.602 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:05.640 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:05.639+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:05.704 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.704 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:05.756 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:05.755+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:05.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:05.815 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.816 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:05.842 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:05.842+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:05.907 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:05.907 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:05.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:05.929 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:05.943 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:05.943+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:05.951 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:05.968 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:05.968 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:05.972 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:05.972 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:05.976 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:05.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:06.014 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:06.014 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:06.036 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:06.036 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:06.044 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:06.043+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:06.056 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:06.057 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T13:50:06.068 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:06.066+0000 7f3c37938640 -1 osd.5 34 handle_osd_map injecting map crc failure 2024-09-13T13:50:06.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:50:06.083 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:06.096 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.096 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:06.096 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:06.096 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.104 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:06.104 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:06.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:06.125 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:06.125 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:06.145 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:06.144+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:06.147 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:06.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:06.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:06.167+0000 7f7e1e160640 0 -- 172.21.15.27:0/1464003008 >> v2:172.21.15.27:6800/2410035394 conn(0x7f7e18219ae0 msgr2=0x7f7e1821bf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:06.173 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:06.178 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:06.177+0000 7f586a715640 -1 osd.9 34 handle_osd_map injecting map crc failure 2024-09-13T13:50:06.196 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:06.196 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:06.210 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.210 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:06.210 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:06.210 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:06.210 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.217 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:06.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:06.217+0000 7f7e1e160640 0 -- 172.21.15.27:0/1464003008 >> v2:172.21.15.159:6805/4073091144 conn(0x7f7e18220df0 msgr2=0x7f7e182231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:06.219 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:06.246 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:06.245+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:06.321 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.321 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:06.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:06.321 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [] 2024-09-13T13:50:06.322 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:06.347 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:06.346+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:06.434 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.434 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:06.434 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:06.434 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:06.486 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:06.485+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:06.545 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.545 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:06.545 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:06.545 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:06.546 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.549 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:06.548+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:06.553 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:06.650 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:06.649+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:06.699 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.699 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:06.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:06.699 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:06.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 18.474613316999999, 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31765422599999998, 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222555+0000", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3489999999999999e-06 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222562+0000", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222572+0000", 2024-09-13T13:50:06.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.031e-05 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222600+0000", 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7647000000000001e-05 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222658+0000", 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8520000000000002e-05 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222671+0000", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2588e-05 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538469+0000", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31579794900000002 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538626+0000", 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000157496 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538781+0000", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015506799999999999 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.539522+0000", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074055599999999996 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540140+0000", 2024-09-13T13:50:06.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061783300000000003 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540177+0000", 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7011999999999997e-05 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540208+0000", 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0735999999999998e-05 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 18.155496793000001, 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:50:06.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:50:06.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:50:06.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497662+0000", 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199504097999998, 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014985478999999999, 2024-09-13T13:50:06.713 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 69 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497662+0000", 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497662+0000", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497663+0000", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6100000000000001e-07 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497739+0000", 2024-09-13T13:50:06.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5804999999999997e-05 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497746+0000", 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7059999999999998e-06 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497751+0000", 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0429999999999998e-06 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.502611+0000", 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048602020000000001 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.502664+0000", 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3328999999999997e-05 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.502747+0000", 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2993999999999999e-05 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512382+0000", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096353029999999996 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512575+0000", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019223800000000001 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512615+0000", 2024-09-13T13:50:06.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0472000000000001e-05 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512648+0000", 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2826e-05 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497762+0000", 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199404752, 2024-09-13T13:50:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015013142, 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497762+0000", 2024-09-13T13:50:06.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497762+0000", 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497762+0000", 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2699999999999999e-07 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497768+0000", 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7150000000000003e-06 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497774+0000", 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497779+0000", 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0769999999999998e-06 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503260+0000", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054804850000000002 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503337+0000", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6987e-05 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503464+0000", 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127167 2024-09-13T13:50:06.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512656+0000", 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0091917089999999993 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512717+0000", 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1692000000000005e-05 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512750+0000", 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2228999999999999e-05 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512775+0000", 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5468000000000001e-05 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497789+0000", 2024-09-13T13:50:06.725 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199377309999999, 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015133669000000001, 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 71 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497789+0000", 2024-09-13T13:50:06.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497789+0000", 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497790+0000", 2024-09-13T13:50:06.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8500000000000001e-07 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497794+0000", 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7419999999999999e-06 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497801+0000", 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.08e-06 2024-09-13T13:50:06.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497805+0000", 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4880000000000001e-06 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503827+0000", 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0060218829999999996 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503884+0000", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6833000000000003e-05 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.503964+0000", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9764000000000004e-05 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512661+0000", 2024-09-13T13:50:06.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0086973499999999995 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512825+0000", 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000164337 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512884+0000", 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8752999999999998e-05 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512923+0000", 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8853999999999998e-05 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497814+0000", 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199352664999999, 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015194713, 2024-09-13T13:50:06.732 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 72 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497814+0000", 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497814+0000", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497814+0000", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2799999999999996e-07 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497893+0000", 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8558000000000002e-05 2024-09-13T13:50:06.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497902+0000", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4010000000000003e-06 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497910+0000", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7819999999999994e-06 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.504253+0000", 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063424629999999996 2024-09-13T13:50:06.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.504330+0000", 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7090999999999998e-05 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.504458+0000", 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127947 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512666+0000", 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082085660000000005 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512959+0000", 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00029268799999999999 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512986+0000", 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7127e-05 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513009+0000", 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2561999999999998e-05 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497925+0000", 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199241113999999, 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015215164999999999, 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 73 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.739 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497925+0000", 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497925+0000", 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497926+0000", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497932+0000", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.327e-06 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497940+0000", 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5029999999999997e-06 2024-09-13T13:50:06.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497948+0000", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9149999999999996e-06 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505124+0000", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071763479999999999 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505177+0000", 2024-09-13T13:50:06.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.223e-05 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505252+0000", 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5692999999999995e-05 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512672+0000", 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074197050000000004 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513055+0000", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00038335099999999998 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513104+0000", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8483000000000003e-05 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513141+0000", 2024-09-13T13:50:06.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6810999999999997e-05 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497959+0000", 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199207286, 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015265496, 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.745 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 74 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497959+0000", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497959+0000", 2024-09-13T13:50:06.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497960+0000", 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497964+0000", 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.721e-06 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497975+0000", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1314999999999999e-05 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497981+0000", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9750000000000004e-06 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505590+0000", 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076087840000000004 2024-09-13T13:50:06.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505637+0000", 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7154000000000003e-05 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.505711+0000", 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3943000000000007e-05 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512678+0000", 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069667339999999996 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513179+0000", 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050122100000000004 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513204+0000", 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5701999999999999e-05 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513225+0000", 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0318000000000001e-05 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199171186000001, 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015359846, 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.751 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 75 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497996+0000", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498055+0000", 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9426999999999999e-05 2024-09-13T13:50:06.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498065+0000", 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8330000000000006e-06 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498072+0000", 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4080000000000003e-06 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506034+0000", 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079622449999999997 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506082+0000", 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999998e-05 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506234+0000", 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015228400000000001 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512684+0000", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064494230000000001 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513270+0000", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058655800000000004 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513321+0000", 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0655000000000002e-05 2024-09-13T13:50:06.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513355+0000", 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4452999999999997e-05 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.498085+0000", 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.199081426999999, 2024-09-13T13:50:06.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015347535000000001, 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 76 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498085+0000", 2024-09-13T13:50:06.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498085+0000", 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498086+0000", 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5600000000000005e-07 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498091+0000", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8330000000000001e-06 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498097+0000", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6880000000000002e-06 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498102+0000", 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1429999999999997e-06 2024-09-13T13:50:06.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.509234+0000", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011131347999999999 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.509283+0000", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9533000000000002e-05 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.509363+0000", 2024-09-13T13:50:06.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9268000000000001e-05 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512936+0000", 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003573672 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513387+0000", 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045028499999999998 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513413+0000", 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6546e-05 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513433+0000", 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9562999999999999e-05 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.763 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.505080432, 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016788053000000001, 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 93 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.764 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192087+0000", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7299999999999996e-07 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192165+0000", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8444999999999999e-05 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192171+0000", 2024-09-13T13:50:06.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8919999999999997e-06 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192175+0000", 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.929e-06 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199613+0000", 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074384250000000002 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199676+0000", 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2170999999999998e-05 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199782+0000", 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010671900000000001 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203302+0000", 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035201080000000001 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208790+0000", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054877570000000002 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208822+0000", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1498999999999998e-05 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208874+0000", 2024-09-13T13:50:06.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2534999999999998e-05 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504984804999999, 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017092758, 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 94 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6400000000000003e-07 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192185+0000", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5359999999999999e-06 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192188+0000", 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2179999999999999e-06 2024-09-13T13:50:06.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192191+0000", 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.762e-06 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202584+0000", 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010393502000000001 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202639+0000", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4867999999999997e-05 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202723+0000", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4164000000000003e-05 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206571+0000", 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003847732 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209193+0000", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026220219999999999 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209224+0000", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1133999999999997e-05 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209275+0000", 2024-09-13T13:50:06.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0355999999999997e-05 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504969290999998, 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017183035999999999, 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 95 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192198+0000", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2000000000000001e-07 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192201+0000", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192204+0000", 2024-09-13T13:50:06.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.095e-06 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192207+0000", 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.65e-06 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203324+0000", 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011117171 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203380+0000", 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6270000000000002e-05 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203467+0000", 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7224000000000004e-05 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206788+0000", 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033212530000000001 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209318+0000", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002529693 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209353+0000", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4761000000000001e-05 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209380+0000", 2024-09-13T13:50:06.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.741e-05 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504956366000002, 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017280957999999999, 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.782 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 96 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:06.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192254+0000", 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3089e-05 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192258+0000", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6630000000000001e-06 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192262+0000", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.366e-06 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203766+0000", 2024-09-13T13:50:06.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011504491 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203819+0000", 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2469999999999997e-05 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203908+0000", 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9716999999999995e-05 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206791+0000", 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002882655 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209425+0000", 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026337750000000001 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209453+0000", 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8707000000000001e-05 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209491+0000", 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7793999999999998e-05 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504899625, 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017314593999999999, 2024-09-13T13:50:06.788 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 97 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2800000000000002e-07 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192270+0000", 2024-09-13T13:50:06.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3029999999999998e-06 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192274+0000", 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0539999999999996e-06 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192294+0000", 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0446999999999999e-05 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204411+0000", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012116511 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204463+0000", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2281000000000003e-05 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204548+0000", 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4648000000000007e-05 2024-09-13T13:50:06.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206793+0000", 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022457190000000002 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209530+0000", 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002736165 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209561+0000", 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1590000000000001e-05 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209582+0000", 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0448000000000001e-05 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504868339000001, 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017390893000000001, 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.795 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 98 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.29e-07 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192304+0000", 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6339999999999998e-06 2024-09-13T13:50:06.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192307+0000", 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0000000000000001e-06 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192310+0000", 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8990000000000001e-06 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204852+0000", 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012542136000000001 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204907+0000", 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4892999999999998e-05 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204992+0000", 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5216000000000005e-05 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206796+0000", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018036059999999999 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209623+0000", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028274730000000001 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209653+0000", 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0009e-05 2024-09-13T13:50:06.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209689+0000", 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5797999999999998e-05 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504845021000001, 2024-09-13T13:50:06.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017460854000000001, 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 99 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:06.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192405+0000", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2977000000000002e-05 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192412+0000", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2629999999999997e-06 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192417+0000", 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1200000000000004e-06 2024-09-13T13:50:06.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205422+0000", 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013005624 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205472+0000", 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0139999999999998e-05 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205588+0000", 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011588500000000001 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208394+0000", 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028053290000000001 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209732+0000", 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013389560000000001 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209761+0000", 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8484e-05 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209782+0000", 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1484e-05 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.807 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.504742419999999, 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017467890999999999, 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 100 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192425+0000", 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7e-07 2024-09-13T13:50:06.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192427+0000", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3910000000000001e-06 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192430+0000", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2370000000000001e-06 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192433+0000", 2024-09-13T13:50:06.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9720000000000001e-06 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206113+0000", 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013680159000000001 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206164+0000", 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494999999999997e-05 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206250+0000", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6243999999999996e-05 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208666+0000", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00241633 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209815+0000", 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001148502 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209852+0000", 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7502000000000001e-05 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209892+0000", 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9589000000000003e-05 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.813 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.101170904, 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:06.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.819 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.087792514, 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:06.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:06.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:06.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:06.824 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:06.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:06.828 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:06.824+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:06.927 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:06.926+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:06.931 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:06.931 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:06.931 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:06.931 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:06.940 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:07.028 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:07.027+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:50:07.083 DEBUG:teuthology.orchestra.run.smithi159:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-09-13T13:50:07.084 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:07.084 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:07.084 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:07.085 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:07.085 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:07.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:07.117 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight read -> 100 2024-09-13T13:50:07.117 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight write -> 25 2024-09-13T13:50:07.117 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight delete -> 10 2024-09-13T13:50:07.117 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight snap_create -> 10 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight snap_remove -> 10 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight rollback -> 10 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight set_chunk -> 30 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight tier_promote -> 10 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight tier_evict -> 10 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight tier_flush -> 5 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:adding op weight write_excl -> 25 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev) 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout:Configuration: 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout: Number of operations: 1500 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout: Number of objects: 50 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout: Max in flight operations: 16 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout: Object size (in bytes): 4000000 2024-09-13T13:50:07.118 INFO:tasks.rados.rados.0.smithi159.stdout: Write stride min: 400000 2024-09-13T13:50:07.119 INFO:tasks.rados.rados.0.smithi159.stdout: Write stride max: 800000 2024-09-13T13:50:07.137 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:07.136+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:07.196 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:07.196 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:07.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:07.197 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [] 2024-09-13T13:50:07.197 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:07.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:07.231 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:07.229+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:07.331 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:07.329+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:07.349 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:07.350 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:07.350 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:07.350 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:07.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:07.431 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:07.431+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:07.461 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:07.461 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:07.461 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:07.461 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:07.461 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:07.470 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:07.533 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:07.532+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.386037933000001, 2024-09-13T13:50:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:50:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:50:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4290.0:1 2.3 e26/25 ondisk, result = 0)", 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.409946+0000", 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.203720547, 2024-09-13T13:50:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015346199999999999, 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409946+0000", 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409946+0000", 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409947+0000", 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8599999999999998e-07 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409951+0000", 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8340000000000001e-06 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409974+0000", 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2603999999999999e-05 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409981+0000", 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1949999999999997e-06 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410022+0000", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1020999999999999e-05 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410028+0000", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6830000000000002e-06 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410100+0000", 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1439000000000005e-05 2024-09-13T13:50:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:07.627 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:07.628 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:07.633 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:07.633+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:07.734 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:07.733+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:07.745 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:07.745 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:07.745 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:07.745 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:07.754 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:07.834 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:07.833+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:07.858 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:07.858 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:07.859 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:07.859 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:07.859 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:07.867 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:07.935 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:07.934+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:07.971 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:07.972 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:07.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:07.972 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [] 2024-09-13T13:50:07.972 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:07.980 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:08.036 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:08.034+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:08.085 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.085 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.085 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:08.086 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.094 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:08.137 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:08.136+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:08.237 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.237 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.238 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:08.238 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:08.238 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.238 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:08.237+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:08.246 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:08.339 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:08.337+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:08.350 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.350 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:08.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:08.350 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [] 2024-09-13T13:50:08.350 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.358 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:08.443 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:08.441+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:08.503 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.504 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.504 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:08.504 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.512 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:08.553 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:08.551+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:08.641 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:08.641+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:08.655 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.655 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.656 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:08.656 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:08.656 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.664 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:08.743 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:08.742+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:08.767 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.767 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:08.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:08.767 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [] 2024-09-13T13:50:08.768 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.776 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:08.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:08.844 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:08.842+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:08.878 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.879 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:08.879 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.887 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:08.945 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:08.943+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:08.976 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:08.976 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:08.990 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:08.990 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:08.990 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:08.991 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:08.991 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:08.999 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:09.000 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:09.022 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:09.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:09.043 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:09.044 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:09.045 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:09.043+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:09.064 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:09.064 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:09.073 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:09.072+0000 7ffaf0cec640 -1 osd.0 37 handle_osd_map injecting map crc failure 2024-09-13T13:50:09.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:50:09.087 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:09.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 20.561985584999999, 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:50:09.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:50:09.110 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:69 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'47, mlcod=25'42)", 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:49.502849+0000", 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.601150848, 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038959929999999999, 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502849+0000", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502849+0000", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502850+0000", 2024-09-13T13:50:09.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0900000000000001e-07 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502902+0000", 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2482000000000003e-05 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502933+0000", 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1134999999999999e-05 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.502941+0000", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2490000000000004e-06 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.503132+0000", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019097 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.503148+0000", 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6549e-05 2024-09-13T13:50:09.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.506716+0000", 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035682969999999998 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.506745+0000", 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8702000000000001e-05 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:90 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'64, mlcod=25'60)", 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.193392+0000", 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.910608465999999, 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041532959999999999, 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193392+0000", 2024-09-13T13:50:09.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193392+0000", 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193392+0000", 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5499999999999997e-07 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193469+0000", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6630000000000003e-05 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193498+0000", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8731999999999999e-05 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.193504+0000", 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0349999999999998e-06 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.194340+0000", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083604100000000004 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.194356+0000", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6206999999999999e-05 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.197506+0000", 2024-09-13T13:50:09.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031501279999999999 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.197545+0000", 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8668000000000003e-05 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:91 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'65, mlcod=25'60)", 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:09.120 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.902573330999999, 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004785274, 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2900000000000002e-07 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201442+0000", 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4711e-05 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201506+0000", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4028000000000004e-05 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201518+0000", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.149e-05 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201579+0000", 2024-09-13T13:50:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0806e-05 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201600+0000", 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1282000000000001e-05 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206182+0000", 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045820169999999999 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206212+0000", 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0111e-05 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:92 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'66, mlcod=25'60)", 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.902550311999999, 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047825760000000002, 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.125 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201455+0000", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201536+0000", 2024-09-13T13:50:09.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0749999999999998e-05 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201539+0000", 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9840000000000001e-06 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205053+0000", 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035134839999999999 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205066+0000", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3079e-05 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206215+0000", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011490509999999999 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206232+0000", 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7416999999999999e-05 2024-09-13T13:50:09.129 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.902543518999998, 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:50:09.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.901155294999999, 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:09.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:50:09.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.139 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.900453292000002, 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:50:09.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:50:09.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:96 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'63)", 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.899967489000002, 2024-09-13T13:50:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050574289999999996, 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:09.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204033+0000", 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204133+0000", 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9450000000000005e-05 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204163+0000", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0018999999999998e-05 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204168+0000", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5539999999999997e-06 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206261+0000", 2024-09-13T13:50:09.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020930889999999998 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206274+0000", 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3018e-05 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209065+0000", 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027905909999999998 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209090+0000", 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4896999999999999e-05 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:97 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'71, mlcod=25'63)", 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.899359064999999, 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044687770000000002, 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.149 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204642+0000", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204651+0000", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2539999999999998e-06 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204708+0000", 2024-09-13T13:50:09.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6959000000000001e-05 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204719+0000", 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0825999999999999e-05 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206607+0000", 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018877659999999999 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206620+0000", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3434999999999999e-05 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209092+0000", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024711659999999999 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209110+0000", 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8051e-05 2024-09-13T13:50:09.153 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:98 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'72, mlcod=25'63)", 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.205097+0000", 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.89890261, 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040357709999999996, 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205097+0000", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205097+0000", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205098+0000", 2024-09-13T13:50:09.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0399999999999996e-07 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205102+0000", 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3560000000000003e-06 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205130+0000", 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7671e-05 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205135+0000", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.434e-06 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206901+0000", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001765745 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206917+0000", 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5531000000000001e-05 2024-09-13T13:50:09.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209111+0000", 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021941920000000002 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209133+0000", 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337999999999999e-05 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:113 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'83, mlcod=25'80)", 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.595973+0000", 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.508027033000001, 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044034410000000001, 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.595973+0000", 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.595973+0000", 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.595974+0000", 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.595979+0000", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1839999999999998e-06 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596004+0000", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5590000000000001e-05 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596009+0000", 2024-09-13T13:50:09.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0930000000000002e-06 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596062+0000", 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2819999999999999e-05 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596086+0000", 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3428999999999999e-05 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.600328+0000", 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042427070000000001 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.600376+0000", 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8028000000000002e-05 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:114 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'84, mlcod=25'80)", 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.596465+0000", 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.507535074, 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041479430000000003, 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596465+0000", 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596465+0000", 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596465+0000", 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596523+0000", 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8069000000000003e-05 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596544+0000", 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0327999999999999e-05 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.596548+0000", 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5589999999999999e-06 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599000+0000", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024518439999999999 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599021+0000", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0883999999999999e-05 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.600586+0000", 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001564771 2024-09-13T13:50:09.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.600613+0000", 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6985999999999999e-05 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.506850955000001, 2024-09-13T13:50:09.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:50:09.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:116 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'86, mlcod=25'80)", 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.506306515999999, 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051291219999999998, 2024-09-13T13:50:09.173 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:09.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597694+0000", 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597697+0000", 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6440000000000002e-06 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597713+0000", 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5699999999999999e-05 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597718+0000", 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1669999999999998e-06 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599712+0000", 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019937420000000002 2024-09-13T13:50:09.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599730+0000", 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8091e-05 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602796+0000", 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003066306 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602823+0000", 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6112999999999999e-05 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:117 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'87, mlcod=25'80)", 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.598883+0000", 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.505116848, 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039730900000000003, 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.178 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598883+0000", 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598883+0000", 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598883+0000", 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0100000000000001e-07 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598948+0000", 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4176000000000004e-05 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598977+0000", 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9230999999999998e-05 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.598981+0000", 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6009999999999997e-06 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599953+0000", 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00097127599999999998 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599968+0000", 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5437000000000001e-05 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602829+0000", 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028604910000000002 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602856+0000", 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7577e-05 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.182 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.500727152, 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:50:09.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.187 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:122 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'92, mlcod=26'88)", 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 18.499486414, 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045221840000000003, 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3500000000000002e-07 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604518+0000", 2024-09-13T13:50:09.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7440000000000001e-06 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604535+0000", 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7088e-05 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604539+0000", 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.681e-06 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.605822+0000", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012838680000000001 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.605837+0000", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.436e-05 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609005+0000", 2024-09-13T13:50:09.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031678019999999999 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609036+0000", 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1205999999999999e-05 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:139 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'105, mlcod=27'100)", 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:09.192 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 17.055942810000001, 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045833219999999999, 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:09.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048058+0000", 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048062+0000", 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7319999999999998e-06 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048079+0000", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6572e-05 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048085+0000", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048124+0000", 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9725999999999999e-05 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048139+0000", 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4874e-05 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052612+0000", 2024-09-13T13:50:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044729260000000003 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052640+0000", 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8223999999999999e-05 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:140 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'106, mlcod=27'102)", 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:52.048699+0000", 2024-09-13T13:50:09.197 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 17.055301263, 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039568210000000001, 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048699+0000", 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:09.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048699+0000", 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048699+0000", 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.75e-07 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048703+0000", 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.143e-06 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048723+0000", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9142000000000001e-05 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048727+0000", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.985e-06 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.051326+0000", 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025996639999999998 2024-09-13T13:50:09.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.051347+0000", 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0877999999999999e-05 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052642+0000", 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001294723 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:09.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052656+0000", 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3711e-05 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:09.202 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:09.203 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:09.203 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:09.204 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:09.204 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:09.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:09.205 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:09.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:09.206 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:09.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:09.207 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:09.202+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:09.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:09.212 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:09.235 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:09.235 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:09.258 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:09.281 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:09.281 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:09.285+0000 7f15e0e5a640 0 -- 172.21.15.27:0/3237400156 >> v2:172.21.15.122:6808/1255553857 conn(0x7f15dc21fb00 msgr2=0x7f15dc221ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:09.304 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:09.303+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:09.308 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:09.316 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:09.317 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:09.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:09.331 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:09.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:09.395 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:09.393+0000 7fc16a180640 -1 osd.11 37 handle_osd_map injecting map crc failure 2024-09-13T13:50:09.405 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:09.403+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:09.438 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:09.438 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:09.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:09.506 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:09.505+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:09.554 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:09.554 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:09.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:09.607 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:09.606+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:09.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:09.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:09.632 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T13:50:09.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:09.671 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:09.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:09.707 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:09.706+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:09.808 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:09.807+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:09.829 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:09.829 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:09.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:09.909 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:09.908+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "cd1b3cf6-d610-4c64-94c3-504da850dd35", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-09-13T13:49:33.733123+0000", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-09-13T13:50:09.071197+0000", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-09-13T13:49:47.205190+0000", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-09-13T13:49:38.189589+0000", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5806080, 2024-09-13T13:50:09.941 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "nodeep-scrub", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "noscrub", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 6, 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-09-13T13:50:09.942 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 4, 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 16, 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "reef", 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "reef", 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-13T13:49:46.385182+0000", 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-09-13T13:50:09.943 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-09-13T13:50:09.944 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "25", 2024-09-13T13:50:09.945 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-13T13:50:09.946 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-13T13:50:09.947 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-13T13:50:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 15.380000114440918, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 15.380000114440918, 2024-09-13T13:50:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.12999999523162842, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 2, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "rbd", 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-13T13:49:48.650814+0000", 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 8193, 2024-09-13T13:50:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:50:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 8, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 8, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 8, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 8, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 8, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-13T13:50:09.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "28", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 2, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 28, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-13T13:50:09.953 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-13T13:50:09.954 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-13T13:50:09.955 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd": {} 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 3.9700000286102295, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 3.9700000286102295, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.5, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.5, 2024-09-13T13:50:09.956 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 3, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "low_tier", 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-13T13:50:03.138857+0000", 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-13T13:50:09.957 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 4, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 4, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 4, 2024-09-13T13:50:09.958 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 4, 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 4, 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-13T13:50:09.959 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "35", 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-13T13:50:09.960 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:50:09.961 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:50:09.962 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": {}, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 3.9800000190734863, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 3.9800000190734863, 2024-09-13T13:50:09.963 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.43999999761581421, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 1.75, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 1.75, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 4, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "unique_pool_0", 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-13T13:50:04.433048+0000", 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-09-13T13:50:09.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 2, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 16, 2024-09-13T13:50:09.965 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 16, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 16, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 16, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 16, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-13T13:50:09.966 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "38", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-13T13:50:09.967 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-13T13:50:09.968 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-13T13:50:09.969 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "fingerprint_algorithm": "sha256" 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "rados": {} 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.9900000095367432, 2024-09-13T13:50:09.970 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.9900000095367432, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.87999999523162842, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.630000114440918, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.630000114440918, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-13T13:50:09.971 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "1259d7f4-8ac0-4fdc-8473-9bdafc036502", 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-09-13T13:50:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4285669868 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.973 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4285669868 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-09-13T13:50:09.974 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4285669868 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4285669868 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6808/4285669868", 2024-09-13T13:50:09.975 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6809/4285669868", 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6811/4285669868", 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6810/4285669868", 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9035b2e5-ff53-4675-83b8-bf76ab2d3735", 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.976 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6808", 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4081030369 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.977 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6809", 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4081030369 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.978 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6811", 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4081030369 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.979 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6810", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4081030369 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.100:6808/4081030369", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.100:6809/4081030369", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.100:6811/4081030369", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.100:6810/4081030369", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "497bd315-4b6e-4f41-b0de-488a942ac4d9", 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:09.981 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6800", 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2454012715 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6801", 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2454012715 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6803", 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2454012715 2024-09-13T13:50:09.983 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6802", 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2454012715 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.984 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.122:6800/2454012715", 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.122:6801/2454012715", 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.122:6803/2454012715", 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.122:6802/2454012715", 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "024edf52-c84f-4d52-b825-c203ed5aecb6", 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.986 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6801", 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3322176894 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6802", 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3322176894 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6804", 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3322176894 2024-09-13T13:50:09.988 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6803", 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3322176894 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.989 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.159:6801/3322176894", 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.159:6802/3322176894", 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.159:6804/3322176894", 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.159:6803/3322176894", 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-09-13T13:50:09.990 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "c440743d-dac9-4c75-b6b3-94360a48f75c", 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 277461260 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-09-13T13:50:09.992 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 277461260 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 277461260 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.993 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 277461260 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6812/277461260", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6813/277461260", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6815/277461260", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6814/277461260", 2024-09-13T13:50:09.994 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "d8203cad-54c8-43f1-af73-ec199c8a41c9", 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:09.995 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6804", 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3923508322 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.996 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6805", 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3923508322 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6807", 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3923508322 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6806", 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3923508322 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:09.998 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.100:6804/3923508322", 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.100:6805/3923508322", 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.100:6807/3923508322", 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.100:6806/3923508322", 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:09.999 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "d5e9b934-6b1e-4f27-838c-6529f69a5b65", 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.000 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6804", 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1372880522 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.001 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6805", 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1372880522 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6807", 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1372880522 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6806", 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1372880522 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.122:6804/1372880522", 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.122:6805/1372880522", 2024-09-13T13:50:10.003 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1372880522", 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.122:6806/1372880522", 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "46d21c0b-bd67-45c9-8bcc-9e76edd03769", 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.004 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.005 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6813", 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1499158947 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.006 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6814", 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1499158947 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.013 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6816", 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1499158947 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6815", 2024-09-13T13:50:10.014 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1499158947 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.159:6813/1499158947", 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.159:6814/1499158947", 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.159:6816/1499158947", 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.159:6815/1499158947", 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0", 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.016 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 330880331 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.017 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 330880331 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 330880331 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-09-13T13:50:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 330880331 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6804/330880331", 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6805/330880331", 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6807/330880331", 2024-09-13T13:50:10.020 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6806/330880331", 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d", 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.021 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 31, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6812", 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1204111774 2024-09-13T13:50:10.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6813", 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1204111774 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.023 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6815", 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1204111774 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.024 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6814", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1204111774 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.100:6812/1204111774", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.100:6813/1204111774", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.100:6815/1204111774", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.100:6814/1204111774", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.025 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "417658a1-1270-435c-8e2a-b9e4a8b0dba6", 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 33, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.026 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6812", 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2574350402 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.027 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6813", 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2574350402 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6815", 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2574350402 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.028 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6814", 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2574350402 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.122:6812/2574350402", 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.122:6813/2574350402", 2024-09-13T13:50:10.029 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2574350402", 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.122:6814/2574350402", 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "9191abab-883f-40a1-8176-3c2e74d90c07", 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.030 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6809", 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1984230453 2024-09-13T13:50:10.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6810", 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1984230453 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.032 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6812", 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1984230453 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6811", 2024-09-13T13:50:10.033 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1984230453 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.159:6809/1984230453", 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.159:6810/1984230453", 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.159:6812/1984230453", 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.159:6811/1984230453", 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 12, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "b9a2390c-9646-489d-b46b-268aa3e73194", 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 22, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6800", 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2410035394 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2410035394 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2410035394 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.037 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2410035394 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6800/2410035394", 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6801/2410035394", 2024-09-13T13:50:10.038 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6803/2410035394", 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6802/2410035394", 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 13, 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18", 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.039 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 21, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6800", 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 450451503 2024-09-13T13:50:10.040 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6801", 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 450451503 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.041 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6803", 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 450451503 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.042 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.100:6802", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 450451503 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.100:6800/450451503", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.100:6801/450451503", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.100:6803/450451503", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.100:6802/450451503", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 14, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "99ad3153-7823-4dd9-9c09-044eba14bcd5", 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 34, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.044 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6808", 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1255553857 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.045 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6809", 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1255553857 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6811", 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1255553857 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.122:6810", 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1255553857 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.122:6808/1255553857", 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.122:6809/1255553857", 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.122:6811/1255553857", 2024-09-13T13:50:10.047 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.122:6810/1255553857", 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 15, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "0880b9c1-ac74-4794-b193-f21d0f1a27f6", 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-13T13:50:10.048 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 0, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 0, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 23, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 0, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 0, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6805", 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4073091144 2024-09-13T13:50:10.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6806", 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4073091144 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-13T13:50:10.050 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6808", 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4073091144 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v2", 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.159:6807", 2024-09-13T13:50:10.051 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4073091144 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.159:6805/4073091144", 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.159:6806/4073091144", 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.159:6808/4073091144", 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.159:6807/4073091144", 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.052 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.512922+0000", 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.987145+0000", 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.054 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.735970+0000", 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.055 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.480014+0000", 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 4, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.954058+0000", 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 5, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.726760+0000", 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 6, 2024-09-13T13:50:10.057 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.911654+0000", 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 7, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.058 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.913247+0000", 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 8, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.205344+0000", 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 9, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.924694+0000", 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.060 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 10, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.492471+0000", 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 11, 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.061 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.816808+0000", 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 12, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.062 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.278234+0000", 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 13, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:43.953059+0000", 2024-09-13T13:50:10.063 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 14, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:45.050800+0000", 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 15, 2024-09-13T13:50:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-13T13:49:44.586118+0000", 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [ 2024-09-13T13:50:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "pgid": "2.2", 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_osd": 9 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": {}, 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-09-13T13:50:10.066 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-09-13T13:50:10.067 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-09-13T13:50:10.068 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-09-13T13:50:10.068 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:10.068 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:10.069 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.069 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.070 INFO:tasks.thrashosds.thrasher:j is {'epoch': 38, 'fsid': 'cd1b3cf6-d610-4c64-94c3-504da850dd35', 'created': '2024-09-13T13:49:33.733123+0000', 'modified': '2024-09-13T13:50:09.071197+0000', 'last_up_change': '2024-09-13T13:49:47.205190+0000', 'last_in_change': '2024-09-13T13:49:38.189589+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 16, 'require_min_compat_client': 'reef', 'min_compat_client': 'reef', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T13:49:46.385182+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T13:49:48.650814+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-09-13T13:50:03.138857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9800000190734863, 'score_stable': 3.9800000190734863, 'optimal_score': 0.4399999976158142, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T13:50:04.433048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1259d7f4-8ac0-4fdc-8473-9bdafc036502', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6808', 'nonce': 4285669868}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6809', 'nonce': 4285669868}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6811', 'nonce': 4285669868}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6810', 'nonce': 4285669868}]}, 'public_addr': '172.21.15.27:6808/4285669868', 'cluster_addr': '172.21.15.27:6809/4285669868', 'heartbeat_back_addr': '172.21.15.27:6811/4285669868', 'heartbeat_front_addr': '172.21.15.27:6810/4285669868', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9035b2e5-ff53-4675-83b8-bf76ab2d3735', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 4081030369}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6809', 'nonce': 4081030369}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6811', 'nonce': 4081030369}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 4081030369}]}, 'public_addr': '172.21.15.100:6808/4081030369', 'cluster_addr': '172.21.15.100:6809/4081030369', 'heartbeat_back_addr': '172.21.15.100:6811/4081030369', 'heartbeat_front_addr': '172.21.15.100:6810/4081030369', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '497bd315-4b6e-4f41-b0de-488a942ac4d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 2454012715}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6801', 'nonce': 2454012715}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6803', 'nonce': 2454012715}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 2454012715}]}, 'public_addr': '172.21.15.122:6800/2454012715', 'cluster_addr': '172.21.15.122:6801/2454012715', 'heartbeat_back_addr': '172.21.15.122:6803/2454012715', 'heartbeat_front_addr': '172.21.15.122:6802/2454012715', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '024edf52-c84f-4d52-b825-c203ed5aecb6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6801', 'nonce': 3322176894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6802', 'nonce': 3322176894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6804', 'nonce': 3322176894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6803', 'nonce': 3322176894}]}, 'public_addr': '172.21.15.159:6801/3322176894', 'cluster_addr': '172.21.15.159:6802/3322176894', 'heartbeat_back_addr': '172.21.15.159:6804/3322176894', 'heartbeat_front_addr': '172.21.15.159:6803/3322176894', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c440743d-dac9-4c75-b6b3-94360a48f75c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6812', 'nonce': 277461260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6813', 'nonce': 277461260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6815', 'nonce': 277461260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6814', 'nonce': 277461260}]}, 'public_addr': '172.21.15.27:6812/277461260', 'cluster_addr': '172.21.15.27:6813/277461260', 'heartbeat_back_addr': '172.21.15.27:6815/277461260', 'heartbeat_front_addr': '172.21.15.27:6814/277461260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'd8203cad-54c8-43f1-af73-ec199c8a41c9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3923508322}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6805', 'nonce': 3923508322}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6807', 'nonce': 3923508322}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 3923508322}]}, 'public_addr': '172.21.15.100:6804/3923508322', 'cluster_addr': '172.21.15.100:6805/3923508322', 'heartbeat_back_addr': '172.21.15.100:6807/3923508322', 'heartbeat_front_addr': '172.21.15.100:6806/3923508322', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd5e9b934-6b1e-4f27-838c-6529f69a5b65', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1372880522}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6805', 'nonce': 1372880522}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6807', 'nonce': 1372880522}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1372880522}]}, 'public_addr': '172.21.15.122:6804/1372880522', 'cluster_addr': '172.21.15.122:6805/1372880522', 'heartbeat_back_addr': '172.21.15.122:6807/1372880522', 'heartbeat_front_addr': '172.21.15.122:6806/1372880522', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '46d21c0b-bd67-45c9-8bcc-9e76edd03769', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6813', 'nonce': 1499158947}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6814', 'nonce': 1499158947}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6816', 'nonce': 1499158947}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6815', 'nonce': 1499158947}]}, 'public_addr': '172.21.15.159:6813/1499158947', 'cluster_addr': '172.21.15.159:6814/1499158947', 'heartbeat_back_addr': '172.21.15.159:6816/1499158947', 'heartbeat_front_addr': '172.21.15.159:6815/1499158947', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6804', 'nonce': 330880331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6805', 'nonce': 330880331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6807', 'nonce': 330880331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6806', 'nonce': 330880331}]}, 'public_addr': '172.21.15.27:6804/330880331', 'cluster_addr': '172.21.15.27:6805/330880331', 'heartbeat_back_addr': '172.21.15.27:6807/330880331', 'heartbeat_front_addr': '172.21.15.27:6806/330880331', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 1204111774}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6813', 'nonce': 1204111774}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6815', 'nonce': 1204111774}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 1204111774}]}, 'public_addr': '172.21.15.100:6812/1204111774', 'cluster_addr': '172.21.15.100:6813/1204111774', 'heartbeat_back_addr': '172.21.15.100:6815/1204111774', 'heartbeat_front_addr': '172.21.15.100:6814/1204111774', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '417658a1-1270-435c-8e2a-b9e4a8b0dba6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 33, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2574350402}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6813', 'nonce': 2574350402}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6815', 'nonce': 2574350402}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2574350402}]}, 'public_addr': '172.21.15.122:6812/2574350402', 'cluster_addr': '172.21.15.122:6813/2574350402', 'heartbeat_back_addr': '172.21.15.122:6815/2574350402', 'heartbeat_front_addr': '172.21.15.122:6814/2574350402', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '9191abab-883f-40a1-8176-3c2e74d90c07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6809', 'nonce': 1984230453}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6810', 'nonce': 1984230453}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6812', 'nonce': 1984230453}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6811', 'nonce': 1984230453}]}, 'public_addr': '172.21.15.159:6809/1984230453', 'cluster_addr': '172.21.15.159:6810/1984230453', 'heartbeat_back_addr': '172.21.15.159:6812/1984230453', 'heartbeat_front_addr': '172.21.15.159:6811/1984230453', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'b9a2390c-9646-489d-b46b-268aa3e73194', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6800', 'nonce': 2410035394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6801', 'nonce': 2410035394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6803', 'nonce': 2410035394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6802', 'nonce': 2410035394}]}, 'public_addr': '172.21.15.27:6800/2410035394', 'cluster_addr': '172.21.15.27:6801/2410035394', 'heartbeat_back_addr': '172.21.15.27:6803/2410035394', 'heartbeat_front_addr': '172.21.15.27:6802/2410035394', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': 'b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6800', 'nonce': 450451503}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6801', 'nonce': 450451503}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6803', 'nonce': 450451503}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 450451503}]}, 'public_addr': '172.21.15.100:6800/450451503', 'cluster_addr': '172.21.15.100:6801/450451503', 'heartbeat_back_addr': '172.21.15.100:6803/450451503', 'heartbeat_front_addr': '172.21.15.100:6802/450451503', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '99ad3153-7823-4dd9-9c09-044eba14bcd5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 1255553857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6809', 'nonce': 1255553857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6811', 'nonce': 1255553857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 1255553857}]}, 'public_addr': '172.21.15.122:6808/1255553857', 'cluster_addr': '172.21.15.122:6809/1255553857', 'heartbeat_back_addr': '172.21.15.122:6811/1255553857', 'heartbeat_front_addr': '172.21.15.122:6810/1255553857', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '0880b9c1-ac74-4794-b193-f21d0f1a27f6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6805', 'nonce': 4073091144}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6806', 'nonce': 4073091144}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6808', 'nonce': 4073091144}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.159:6807', 'nonce': 4073091144}]}, 'public_addr': '172.21.15.159:6805/4073091144', 'cluster_addr': '172.21.15.159:6806/4073091144', 'heartbeat_back_addr': '172.21.15.159:6808/4073091144', 'heartbeat_front_addr': '172.21.15.159:6807/4073091144', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.512922+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.987145+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.735970+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.480014+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.954058+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.726760+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.911654+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.913247+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.205344+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.924694+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.492471+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.816808+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.278234+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:43.953059+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:45.050800+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T13:49:44.586118+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [{'pgid': '2.2', 'primary_osd': 9}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T13:50:10.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:50:10.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:10.072 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:10.068+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:10.094 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:50:10.094 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:10.093+0000 7f586c719640 -1 osd.9 38 handle_osd_map injecting map crc failure 2024-09-13T13:50:10.097 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:10.095+0000 7f1f37764640 -1 osd.4 38 handle_osd_map injecting map crc failure 2024-09-13T13:50:10.169 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:10.168+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:10.204 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.205 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:10.271 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:10.270+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:10.323 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.323 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:10.372 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:10.371+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:10.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:50:10.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-13T13:50:10.365868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":438232,"kb_used_data":5848,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511208,"statfs":{"total":1546188226560,"available":1545739476992,"internally_reserved":0,"allocated":5988352,"data_stored":3454238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":442473553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000860"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.068453+0000","last_change":"2024-09-13T13:50:05.068453+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.068453+0000","last_undegraded":"2024-09-13T13:50:05.068453+0000","last_fullsized":"2024-09-13T13:50:05.068453+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.073185+0000","last_change":"2024-09-13T13:50:06.073186+0000","last_active":"2024-09-13T13:50:07.073185+0000","last_peered":"2024-09-13T13:50:07.073185+0000","last_clean":"2024-09-13T13:50:07.073185+0000","last_became_active":"2024-09-13T13:50:06.072867+0000","last_became_peered":"2024-09-13T13:50:06.072867+0000","last_unstale":"2024-09-13T13:50:07.073185+0000","last_undegraded":"2024-09-13T13:50:07.073185+0000","last_fullsized":"2024-09-13T13:50:07.073185+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:31.662776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.123816+0000","last_change":"2024-09-13T13:50:05.123816+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.123816+0000","last_undegraded":"2024-09-13T13:50:05.123816+0000","last_fullsized":"2024-09-13T13:50:05.123816+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.123834+0000","last_change":"2024-09-13T13:50:05.123834+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.123834+0000","last_undegraded":"2024-09-13T13:50:05.123834+0000","last_fullsized":"2024-09-13T13:50:05.123834+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T13:50:06.076356+0000","last_change":"2024-09-13T13:50:06.073496+0000","last_active":"2024-09-13T13:50:06.076356+0000","last_peered":"2024-09-13T13:50:06.076356+0000","last_clean":"2024-09-13T13:50:06.076356+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:50:06.076356+0000","last_undegraded":"2024-09-13T13:50:06.076356+0000","last_fullsized":"2024-09-13T13:50:06.076356+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:25.107571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.882110+0000","last_change":"2024-09-13T13:50:06.077380+0000","last_active":"2024-09-13T13:50:07.882110+0000","last_peered":"2024-09-13T13:50:07.882110+0000","last_clean":"2024-09-13T13:50:07.882110+0000","last_became_active":"2024-09-13T13:50:06.077071+0000","last_became_peered":"2024-09-13T13:50:06.077071+0000","last_unstale":"2024-09-13T13:50:07.882110+0000","last_undegraded":"2024-09-13T13:50:07.882110+0000","last_fullsized":"2024-09-13T13:50:07.882110+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:09.611931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.883799+0000","last_change":"2024-09-13T13:50:06.072902+0000","last_active":"2024-09-13T13:50:07.883799+0000","last_peered":"2024-09-13T13:50:07.883799+0000","last_clean":"2024-09-13T13:50:07.883799+0000","last_became_active":"2024-09-13T13:50:06.072545+0000","last_became_peered":"2024-09-13T13:50:06.072545+0000","last_unstale":"2024-09-13T13:50:07.883799+0000","last_undegraded":"2024-09-13T13:50:07.883799+0000","last_fullsized":"2024-09-13T13:50:07.883799+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:30.733055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.069343+0000","last_change":"2024-09-13T13:50:05.069343+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.069343+0000","last_undegraded":"2024-09-13T13:50:05.069343+0000","last_fullsized":"2024-09-13T13:50:05.069343+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.124417+0000","last_change":"2024-09-13T13:50:05.124417+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.124417+0000","last_undegraded":"2024-09-13T13:50:05.124417+0000","last_fullsized":"2024-09-13T13:50:05.124417+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":167,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.126560+0000","last_change":"2024-09-13T13:49:48.539255+0000","last_active":"2024-09-13T13:50:05.126560+0000","last_peered":"2024-09-13T13:50:05.126560+0000","last_clean":"2024-09-13T13:50:05.126560+0000","last_became_active":"2024-09-13T13:49:48.538219+0000","last_became_peered":"2024-09-13T13:49:48.538219+0000","last_unstale":"2024-09-13T13:50:05.126560+0000","last_undegraded":"2024-09-13T13:50:05.126560+0000","last_fullsized":"2024-09-13T13:50:05.126560+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:49:47.205190+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.389193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":34,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T13:50:06.070072+0000","last_change":"2024-09-13T13:49:52.369080+0000","last_active":"2024-09-13T13:50:06.070072+0000","last_peered":"2024-09-13T13:50:06.070072+0000","last_clean":"2024-09-13T13:50:06.070072+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:50:06.070072+0000","last_undegraded":"2024-09-13T13:50:06.070072+0000","last_fullsized":"2024-09-13T13:50:06.070072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:13.538100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T13:50:06.070276+0000","last_change":"2024-09-13T13:50:05.071494+0000","last_active":"2024-09-13T13:50:06.070276+0000","last_peered":"2024-09-13T13:50:06.070276+0000","last_clean":"2024-09-13T13:50:06.070276+0000","last_became_active":"2024-09-13T13:50:05.071221+0000","last_became_peered":"2024-09-13T13:50:05.071221+0000","last_unstale":"2024-09-13T13:50:06.070276+0000","last_undegraded":"2024-09-13T13:50:06.070276+0000","last_fullsized":"2024-09-13T13:50:06.070276+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:07.675536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.864767+0000","last_change":"2024-09-13T13:50:06.075847+0000","last_active":"2024-09-13T13:50:07.864767+0000","last_peered":"2024-09-13T13:50:07.864767+0000","last_clean":"2024-09-13T13:50:07.864767+0000","last_became_active":"2024-09-13T13:50:06.075484+0000","last_became_peered":"2024-09-13T13:50:06.075484+0000","last_unstale":"2024-09-13T13:50:07.864767+0000","last_undegraded":"2024-09-13T13:50:07.864767+0000","last_fullsized":"2024-09-13T13:50:07.864767+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:31.188715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.881904+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:50:07.881904+0000","last_peered":"2024-09-13T13:50:07.881904+0000","last_clean":"2024-09-13T13:50:07.881904+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:50:07.881904+0000","last_undegraded":"2024-09-13T13:50:07.881904+0000","last_fullsized":"2024-09-13T13:50:07.881904+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:14.783466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.123912+0000","last_change":"2024-09-13T13:50:05.123338+0000","last_active":"2024-09-13T13:50:05.123912+0000","last_peered":"2024-09-13T13:50:05.123912+0000","last_clean":"2024-09-13T13:50:05.123912+0000","last_became_active":"2024-09-13T13:50:05.123078+0000","last_became_peered":"2024-09-13T13:50:05.123078+0000","last_unstale":"2024-09-13T13:50:05.123912+0000","last_undegraded":"2024-09-13T13:50:05.123912+0000","last_fullsized":"2024-09-13T13:50:05.123912+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:18.652628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.074929+0000","last_change":"2024-09-13T13:50:06.076579+0000","last_active":"2024-09-13T13:50:07.074929+0000","last_peered":"2024-09-13T13:50:07.074929+0000","last_clean":"2024-09-13T13:50:07.074929+0000","last_became_active":"2024-09-13T13:50:06.076355+0000","last_became_peered":"2024-09-13T13:50:06.076355+0000","last_unstale":"2024-09-13T13:50:07.074929+0000","last_undegraded":"2024-09-13T13:50:07.074929+0000","last_fullsized":"2024-09-13T13:50:07.074929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:10.746415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T13:50:06.069973+0000","last_change":"2024-09-13T13:49:52.369120+0000","last_active":"2024-09-13T13:50:06.069973+0000","last_peered":"2024-09-13T13:50:06.069973+0000","last_clean":"2024-09-13T13:50:06.069973+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:50:06.069973+0000","last_undegraded":"2024-09-13T13:50:06.069973+0000","last_fullsized":"2024-09-13T13:50:06.069973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.751922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.399007+0000","last_change":"2024-09-13T13:50:05.397984+0000","last_active":"2024-09-13T13:50:05.399007+0000","last_peered":"2024-09-13T13:50:05.399007+0000","last_clean":"2024-09-13T13:50:05.399007+0000","last_became_active":"2024-09-13T13:50:05.397681+0000","last_became_peered":"2024-09-13T13:50:05.397681+0000","last_unstale":"2024-09-13T13:50:05.399007+0000","last_undegraded":"2024-09-13T13:50:05.399007+0000","last_fullsized":"2024-09-13T13:50:05.399007+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:09.164760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.124433+0000","last_change":"2024-09-13T13:50:05.124433+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.124433+0000","last_undegraded":"2024-09-13T13:50:05.124433+0000","last_fullsized":"2024-09-13T13:50:05.124433+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":32,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.066482+0000","last_change":"2024-09-13T13:50:02.072282+0000","last_active":"2024-09-13T13:50:05.066482+0000","last_peered":"2024-09-13T13:50:05.066482+0000","last_clean":"2024-09-13T13:50:05.066482+0000","last_became_active":"2024-09-13T13:50:02.071999+0000","last_became_peered":"2024-09-13T13:50:02.071999+0000","last_unstale":"2024-09-13T13:50:05.066482+0000","last_undegraded":"2024-09-13T13:50:05.066482+0000","last_fullsized":"2024-09-13T13:50:05.066482+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:58.783364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.075603+0000","last_change":"2024-09-13T13:50:05.074631+0000","last_active":"2024-09-13T13:50:05.075603+0000","last_peered":"2024-09-13T13:50:05.075603+0000","last_clean":"2024-09-13T13:50:05.075603+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:50:05.075603+0000","last_undegraded":"2024-09-13T13:50:05.075603+0000","last_fullsized":"2024-09-13T13:50:05.075603+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:07.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.124443+0000","last_change":"2024-09-13T13:50:05.124443+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.124443+0000","last_undegraded":"2024-09-13T13:50:05.124443+0000","last_fullsized":"2024-09-13T13:50:05.124443+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.074756+0000","last_change":"2024-09-13T13:49:52.370546+0000","last_active":"2024-09-13T13:50:07.074756+0000","last_peered":"2024-09-13T13:50:07.074756+0000","last_clean":"2024-09-13T13:50:07.074756+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:50:07.074756+0000","last_undegraded":"2024-09-13T13:50:07.074756+0000","last_fullsized":"2024-09-13T13:50:07.074756+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:11.318833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.067367+0000","last_change":"2024-09-13T13:50:05.067367+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.067367+0000","last_undegraded":"2024-09-13T13:50:05.067367+0000","last_fullsized":"2024-09-13T13:50:05.067367+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":36,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.864635+0000","last_change":"2024-09-13T13:49:52.239921+0000","last_active":"2024-09-13T13:50:07.864635+0000","last_peered":"2024-09-13T13:50:07.864635+0000","last_clean":"2024-09-13T13:50:07.864635+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:50:07.864635+0000","last_undegraded":"2024-09-13T13:50:07.864635+0000","last_fullsized":"2024-09-13T13:50:07.864635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:52.937080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.121918+0000","last_change":"2024-09-13T13:50:05.121918+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.121918+0000","last_undegraded":"2024-09-13T13:50:05.121918+0000","last_fullsized":"2024-09-13T13:50:05.121918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T13:50:05.066421+0000","last_change":"2024-09-13T13:49:52.248018+0000","last_active":"2024-09-13T13:50:05.066421+0000","last_peered":"2024-09-13T13:50:05.066421+0000","last_clean":"2024-09-13T13:50:05.066421+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:50:05.066421+0000","last_undegraded":"2024-09-13T13:50:05.066421+0000","last_fullsized":"2024-09-13T13:50:05.066421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:50:49.267208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2024-09-13T13:50:05.067369+0000","last_change":"2024-09-13T13:50:05.067369+0000","last_active":"2024-09-13T13:50:05.059988+0000","last_peered":"2024-09-13T13:50:05.059988+0000","last_clean":"2024-09-13T13:50:05.059988+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T13:50:05.067369+0000","last_undegraded":"2024-09-13T13:50:05.067369+0000","last_fullsized":"2024-09-13T13:50:05.067369+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-13T13:50:07.881986+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:50:07.881986+0000","last_peered":"2024-09-13T13:50:07.881986+0000","last_clean":"2024-09-13T13:50:07.881986+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:50:07.881986+0000","last_undegraded":"2024-09-13T13:50:07.881986+0000","last_fullsized":"2024-09-13T13:50:07.881986+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:03.281318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":13},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247816,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":905216,"data_stored":732347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247815,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":352256,"data_stored":153320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":144700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":299008,"data_stored":138974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":138156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":339968,"data_stored":144940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":719712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":315392,"data_stored":139805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":135227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,5,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":136174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":137849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":138438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":352256,"data_stored":155333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":149888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":282624,"data_stored":149697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":139678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:50:10.419 INFO:tasks.thrashosds.thrasher:Setting 4.9 to [3, 13, 11, 15] 2024-09-13T13:50:10.419 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.9', '3', '13', '11', '15'] 2024-09-13T13:50:10.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.9 3 13 11 15 2024-09-13T13:50:10.441 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.441 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.450 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:10.473 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:10.472+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:10.574 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:10.573+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:10.602 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.602 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:10.674 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:10.674+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:10.718 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.719 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:10.776 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:10.775+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:10.835 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.835 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:10.877 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:10.876+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:10.959 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:10.959 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:10.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:10.978 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:10.977+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:11.071 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.072 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.079 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:11.078+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:11.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:11.102 INFO:teuthology.orchestra.run.smithi027.stderr:set 4.9 pg_upmap_items mapping to [3->13,11->15] 2024-09-13T13:50:11.180 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:11.180+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:11.184 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.184 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:11.282 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:11.281+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:11.295 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.295 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:11.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:11.383 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:11.382+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:11.406 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.406 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:11.484 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:11.483+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:11.486 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:11.487 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:11.509 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:11.509 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:11.518 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.518 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:11.531 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:11.553 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:11.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:11.574 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:11.588 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:11.584+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:11.595 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:11.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:11.617 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:11.617 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:11.631 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:11.631 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:11.638 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:11.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:11.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:11.660 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:11.660 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:11.681 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:11.686 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:11.685+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:11.703 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:11.703 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:11.725 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:11.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:11.745 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:11.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:11.785 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:11.785 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:11.785 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:11.785 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:11.785 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:11.786 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:11.785+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:11.788 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:11.788 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:11.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:11.787+0000 7f92fb59e640 0 -- 172.21.15.27:0/2608533871 >> v2:172.21.15.159:6805/4073091144 conn(0x7f92fc2226a0 msgr2=0x7f92fc224a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:50:11.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:11.887 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:11.886+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:11.936 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 20.702028912999999, 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:11.937 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:50:11.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:11.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:11.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 19.693994767, 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:11.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:50:11.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:50:11.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:50:11.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:11.947 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:11.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:11.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:11.947 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:11.947 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:11.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:11.988 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:11.987+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:12.011 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:12.011 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:12.050 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:12.051 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:12.051 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:12.051 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:12.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:12.089 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:12.088+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:12.104 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:12.103+0000 7f586c719640 -1 osd.9 40 handle_osd_map injecting map crc failure 2024-09-13T13:50:12.107 INFO:tasks.rados.rados.0.smithi159.stdout:1: write initial oid 1 2024-09-13T13:50:12.107 INFO:tasks.rados.rados.0.smithi159.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-13T13:50:12.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:50:12.112 INFO:tasks.rados.rados.0.smithi159.stdout:1: writing smithi15938319-1 from 416807 to 892056 tid 1 2024-09-13T13:50:12.119 INFO:tasks.rados.rados.0.smithi159.stdout:1: writing smithi15938319-1 from 1542129 to 2016807 tid 2 2024-09-13T13:50:12.119 INFO:tasks.rados.rados.0.smithi159.stdout:2: write initial oid 2 2024-09-13T13:50:12.119 INFO:tasks.rados.rados.0.smithi159.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-13T13:50:12.124 INFO:tasks.rados.rados.0.smithi159.stdout:2: writing smithi15938319-2 from 416807 to 892056 tid 1 2024-09-13T13:50:12.130 INFO:tasks.rados.rados.0.smithi159.stdout:2: writing smithi15938319-2 from 1542129 to 2016807 tid 2 2024-09-13T13:50:12.130 INFO:tasks.rados.rados.0.smithi159.stdout:3: write initial oid 3 2024-09-13T13:50:12.130 INFO:tasks.rados.rados.0.smithi159.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-13T13:50:12.135 INFO:tasks.rados.rados.0.smithi159.stdout:3: writing smithi15938319-3 from 433614 to 984112 tid 1 2024-09-13T13:50:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:3: writing smithi15938319-3 from 1600611 to 2033614 tid 2 2024-09-13T13:50:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:4: write initial oid 4 2024-09-13T13:50:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-13T13:50:12.147 INFO:tasks.rados.rados.0.smithi159.stdout:4: writing smithi15938319-4 from 450421 to 1076168 tid 1 2024-09-13T13:50:12.152 INFO:tasks.rados.rados.0.smithi159.stdout:4: writing smithi15938319-4 from 1659093 to 2050421 tid 2 2024-09-13T13:50:12.152 INFO:tasks.rados.rados.0.smithi159.stdout:5: write initial oid 5 2024-09-13T13:50:12.152 INFO:tasks.rados.rados.0.smithi159.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-13T13:50:12.158 INFO:tasks.rados.rados.0.smithi159.stdout:5: writing smithi15938319-5 from 467228 to 1168224 tid 1 2024-09-13T13:50:12.163 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:12.163 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:12.163 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:12.163 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:12.163 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:12.164 INFO:tasks.rados.rados.0.smithi159.stdout:5: writing smithi15938319-5 from 1717575 to 2067228 tid 2 2024-09-13T13:50:12.164 INFO:tasks.rados.rados.0.smithi159.stdout:6: write initial oid 6 2024-09-13T13:50:12.164 INFO:tasks.rados.rados.0.smithi159.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-13T13:50:12.170 INFO:tasks.rados.rados.0.smithi159.stdout:6: writing smithi15938319-6 from 484035 to 1260280 tid 1 2024-09-13T13:50:12.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:12.176 INFO:tasks.rados.rados.0.smithi159.stdout:6: writing smithi15938319-6 from 2059704 to 2084035 tid 2 2024-09-13T13:50:12.176 INFO:tasks.rados.rados.0.smithi159.stdout:7: write initial oid 7 2024-09-13T13:50:12.176 INFO:tasks.rados.rados.0.smithi159.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-13T13:50:12.179 INFO:tasks.rados.rados.0.smithi159.stdout:7: writing smithi15938319-7 from 500842 to 952336 tid 1 2024-09-13T13:50:12.186 INFO:tasks.rados.rados.0.smithi159.stdout:7: writing smithi15938319-7 from 1718186 to 2100842 tid 2 2024-09-13T13:50:12.186 INFO:tasks.rados.rados.0.smithi159.stdout:8: write initial oid 8 2024-09-13T13:50:12.186 INFO:tasks.rados.rados.0.smithi159.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-13T13:50:12.189 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:12.188+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:8: writing smithi15938319-8 from 517649 to 1044392 tid 1 2024-09-13T13:50:12.198 INFO:tasks.rados.rados.0.smithi159.stdout:8: writing smithi15938319-8 from 1776668 to 2117649 tid 2 2024-09-13T13:50:12.198 INFO:tasks.rados.rados.0.smithi159.stdout:9: write initial oid 9 2024-09-13T13:50:12.198 INFO:tasks.rados.rados.0.smithi159.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-13T13:50:12.204 INFO:tasks.rados.rados.0.smithi159.stdout:9: writing smithi15938319-9 from 534456 to 1252801 tid 1 2024-09-13T13:50:12.209 INFO:tasks.rados.rados.0.smithi159.stdout:9: writing smithi15938319-9 from 1951503 to 2134456 tid 2 2024-09-13T13:50:12.209 INFO:tasks.rados.rados.0.smithi159.stdout:10: write initial oid 10 2024-09-13T13:50:12.210 INFO:tasks.rados.rados.0.smithi159.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-13T13:50:12.217 INFO:tasks.rados.rados.0.smithi159.stdout:10: writing smithi15938319-10 from 551263 to 1344857 tid 1 2024-09-13T13:50:12.221 INFO:tasks.rados.rados.0.smithi159.stdout:10: writing smithi15938319-10 from 1893632 to 2151263 tid 2 2024-09-13T13:50:12.221 INFO:tasks.rados.rados.0.smithi159.stdout:11: write initial oid 11 2024-09-13T13:50:12.221 INFO:tasks.rados.rados.0.smithi159.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-13T13:50:12.227 INFO:tasks.rados.rados.0.smithi159.stdout:11: writing smithi15938319-11 from 568070 to 1036913 tid 1 2024-09-13T13:50:12.233 INFO:tasks.rados.rados.0.smithi159.stdout:11: writing smithi15938319-11 from 1552114 to 2168070 tid 2 2024-09-13T13:50:12.233 INFO:tasks.rados.rados.0.smithi159.stdout:12: write initial oid 12 2024-09-13T13:50:12.233 INFO:tasks.rados.rados.0.smithi159.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-13T13:50:12.239 INFO:tasks.rados.rados.0.smithi159.stdout:12: writing smithi15938319-12 from 584877 to 1128969 tid 1 2024-09-13T13:50:12.245 INFO:tasks.rados.rados.0.smithi159.stdout:12: writing smithi15938319-12 from 1610596 to 2172599 tid 2 2024-09-13T13:50:12.245 INFO:tasks.rados.rados.0.smithi159.stdout:13: write initial oid 13 2024-09-13T13:50:12.245 INFO:tasks.rados.rados.0.smithi159.stdout:1: finishing write tid 1 to smithi15938319-1 2024-09-13T13:50:12.245 INFO:tasks.rados.rados.0.smithi159.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-13T13:50:12.251 INFO:tasks.rados.rados.0.smithi159.stdout:13: writing smithi15938319-13 from 601684 to 1221025 tid 1 2024-09-13T13:50:12.257 INFO:tasks.rados.rados.0.smithi159.stdout:13: writing smithi15938319-13 from 1669078 to 2201684 tid 2 2024-09-13T13:50:12.257 INFO:tasks.rados.rados.0.smithi159.stdout:14: write initial oid 14 2024-09-13T13:50:12.257 INFO:tasks.rados.rados.0.smithi159.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-13T13:50:12.264 INFO:tasks.rados.rados.0.smithi159.stdout:14: writing smithi15938319-14 from 618491 to 1313081 tid 1 2024-09-13T13:50:12.269 INFO:tasks.rados.rados.0.smithi159.stdout:14: writing smithi15938319-14 from 2011207 to 2218491 tid 2 2024-09-13T13:50:12.269 INFO:tasks.rados.rados.0.smithi159.stdout:15: write initial oid 15 2024-09-13T13:50:12.269 INFO:tasks.rados.rados.0.smithi159.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-13T13:50:12.276 INFO:tasks.rados.rados.0.smithi159.stdout:15: writing smithi15938319-15 from 635298 to 1405137 tid 1 2024-09-13T13:50:12.278 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:12.278 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:41 4.f 4:fec51f4e:::smithi15938319-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.045882994000000003, 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087181849999999998, 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 41 2024-09-13T13:50:12.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231914+0000", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1869999999999996e-06 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232499+0000", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00058453499999999996 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232510+0000", 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0796e-05 2024-09-13T13:50:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232519+0000", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7389999999999994e-06 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232585+0000", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5976000000000007e-05 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232590+0000", 2024-09-13T13:50:12.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.234923+0000", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023328820000000001 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.235015+0000", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2182999999999995e-05 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.235114+0000", 2024-09-13T13:50:12.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8932999999999996e-05 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240135+0000", 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050208220000000003 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240564+0000", 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000429694 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240597+0000", 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2839000000000002e-05 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240627+0000", 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0127000000000001e-05 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.285 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:42 4.f 4:fec51f4e:::smithi15938319-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.044639363000000001, 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010249345999999999, 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.286 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 42 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2399999999999998e-07 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233820+0000", 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066637700000000001 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233829+0000", 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7919999999999998e-06 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233836+0000", 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9600000000000003e-06 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233886+0000", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0929e-05 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233891+0000", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4800000000000003e-06 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.238648+0000", 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047567679999999998 2024-09-13T13:50:12.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239101+0000", 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000452872 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239171+0000", 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0440999999999999e-05 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240313+0000", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011419170000000001 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243338+0000", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030247899999999999 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243370+0000", 2024-09-13T13:50:12.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2308999999999998e-05 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243402+0000", 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2187000000000002e-05 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:43 4.f 4:fec51f4e:::smithi15938319-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:12.293 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.043942044, 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097138150000000006, 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 43 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233851+0000", 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4499999999999997e-07 2024-09-13T13:50:12.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233853+0000", 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2869999999999999e-06 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233862+0000", 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0459999999999994e-06 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233866+0000", 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0550000000000001e-06 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233913+0000", 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7360000000000001e-05 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233916+0000", 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7070000000000001e-06 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239867+0000", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059507259999999999 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239925+0000", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8322999999999997e-05 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240006+0000", 2024-09-13T13:50:12.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0562000000000006e-05 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240882+0000", 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00087657100000000003 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243503+0000", 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002620603 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243530+0000", 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.673e-05 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243564+0000", 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.43e-05 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.300 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:44 4.f 4:fec51f4e:::smithi15938319-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.043908590999999997, 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098119149999999992, 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:12.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 44 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3500000000000002e-07 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233887+0000", 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7020000000000001e-06 2024-09-13T13:50:12.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233890+0000", 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6669999999999998e-06 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233893+0000", 2024-09-13T13:50:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1310000000000002e-06 2024-09-13T13:50:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233935+0000", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1909000000000001e-05 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233939+0000", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.721e-06 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240164+0000", 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062245 2024-09-13T13:50:12.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240207+0000", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3377999999999997e-05 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243598+0000", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033911689999999999 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243637+0000", 2024-09-13T13:50:12.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9276000000000001e-05 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243696+0000", 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8127000000000001e-05 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:41 4.f e41/34 ondisk, result = 0)", 2024-09-13T13:50:12.308 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.240489+0000", 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.037303485999999997, 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014501800000000001, 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240489+0000", 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240489+0000", 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240490+0000", 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.062e-06 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240494+0000", 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3390000000000003e-06 2024-09-13T13:50:12.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240514+0000", 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0029000000000001e-05 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240520+0000", 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1199999999999999e-06 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240557+0000", 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6980000000000002e-05 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240562+0000", 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3610000000000003e-06 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240634+0000", 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2126999999999996e-05 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:42 4.f e41/34 ondisk, result = 0)", 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.243248+0000", 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.034544006000000002, 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000158629, 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243248+0000", 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243248+0000", 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243249+0000", 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.54e-07 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243254+0000", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.729e-06 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243279+0000", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5601000000000001e-05 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243286+0000", 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.207e-06 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243330+0000", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4759000000000003e-05 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243335+0000", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5419999999999999e-06 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243407+0000", 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2037000000000002e-05 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.316 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:43 4.f e41/34 ondisk, result = 0)", 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.243429+0000", 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.034363496, 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014216700000000001, 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243429+0000", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243429+0000", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243429+0000", 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4400000000000001e-07 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243432+0000", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8030000000000001e-06 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243447+0000", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.517e-05 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243454+0000", 2024-09-13T13:50:12.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8530000000000003e-06 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243492+0000", 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7796000000000002e-05 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243498+0000", 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5719999999999999e-06 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:12.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243571+0000", 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2528999999999996e-05 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:12.321 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:12.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:12.325 INFO:tasks.rados.rados.0.smithi159.stdout:15: writing smithi15938319-15 from 2069689 to 2235298 tid 2 2024-09-13T13:50:12.325 INFO:tasks.rados.rados.0.smithi159.stdout:16: write initial oid 16 2024-09-13T13:50:12.325 INFO:tasks.rados.rados.0.smithi159.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-13T13:50:12.325 INFO:tasks.rados.rados.0.smithi159.stdout:16: writing smithi15938319-16 from 652105 to 1097193 tid 1 2024-09-13T13:50:12.325 INFO:tasks.rados.rados.0.smithi159.stdout:16: writing smithi15938319-16 from 1728171 to 2181159 tid 2 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: finishing write tid 2 to smithi15938319-1 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: finishing write tid 3 to smithi15938319-1 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: finishing write tid 4 to smithi15938319-1 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: oid 1 updating version 0 to 1 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: oid 1 updating version 1 to 3 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: oid 1 version 3 is already newer than 2 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:2: finishing write tid 1 to smithi15938319-2 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:2: finishing write tid 2 to smithi15938319-2 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:2: finishing write tid 3 to smithi15938319-2 2024-09-13T13:50:12.326 INFO:tasks.rados.rados.0.smithi159.stdout:1: done (15 left) 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:17: write initial oid 17 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:17: writing smithi15938319-17 from 668912 to 1305602 tid 1 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:17: writing smithi15938319-17 from 1903006 to 2268912 tid 2 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: finishing write tid 4 to smithi15938319-2 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: oid 2 updating version 0 to 1 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: oid 2 updating version 1 to 3 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: oid 2 version 3 is already newer than 2 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:3: finishing write tid 1 to smithi15938319-3 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:3: finishing write tid 2 to smithi15938319-3 2024-09-13T13:50:12.327 INFO:tasks.rados.rados.0.smithi159.stdout:2: done (15 left) 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:18: write initial oid 18 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:18: writing smithi15938319-18 from 685719 to 1397658 tid 1 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:18: writing smithi15938319-18 from 1845135 to 2285719 tid 2 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: finishing write tid 3 to smithi15938319-3 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: finishing write tid 4 to smithi15938319-3 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: oid 3 updating version 0 to 3 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: oid 3 version 3 is already newer than 2 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: oid 3 version 3 is already newer than 1 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:12.328 INFO:tasks.rados.rados.0.smithi159.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:12.329 INFO:tasks.rados.rados.0.smithi159.stdout:4: finishing write tid 1 to smithi15938319-4 2024-09-13T13:50:12.329 INFO:tasks.rados.rados.0.smithi159.stdout:4: finishing write tid 2 to smithi15938319-4 2024-09-13T13:50:12.329 INFO:tasks.rados.rados.0.smithi159.stdout:3: done (15 left) 2024-09-13T13:50:12.329 INFO:tasks.rados.rados.0.smithi159.stdout:19: write initial oid 19 2024-09-13T13:50:12.329 INFO:tasks.rados.rados.0.smithi159.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-13T13:50:12.329 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:12.321+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:12.330 INFO:tasks.rados.rados.0.smithi159.stdout:19: writing smithi15938319-19 from 702526 to 1489714 tid 1 2024-09-13T13:50:12.330 INFO:tasks.rados.rados.0.smithi159.stdout:19: writing smithi15938319-19 from 1903617 to 2302526 tid 2 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: finishing write tid 3 to smithi15938319-4 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: finishing write tid 4 to smithi15938319-4 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: oid 4 updating version 0 to 6 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: oid 4 version 6 is already newer than 5 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: oid 4 version 6 is already newer than 4 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:5: finishing write tid 1 to smithi15938319-5 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:5: finishing write tid 2 to smithi15938319-5 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:7: finishing write tid 1 to smithi15938319-7 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:6: finishing write tid 1 to smithi15938319-6 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:6: finishing write tid 2 to smithi15938319-6 2024-09-13T13:50:12.331 INFO:tasks.rados.rados.0.smithi159.stdout:6: finishing write tid 3 to smithi15938319-6 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: finishing write tid 4 to smithi15938319-6 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: oid 6 updating version 0 to 2 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: oid 6 updating version 2 to 3 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: oid 6 version 3 is already newer than 1 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:4: done (15 left) 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:6: done (14 left) 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:20: write initial oid 20 2024-09-13T13:50:12.332 INFO:tasks.rados.rados.0.smithi159.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-13T13:50:12.336 INFO:tasks.rados.rados.0.smithi159.stdout:20: writing smithi15938319-20 from 719333 to 1181770 tid 1 2024-09-13T13:50:12.342 INFO:tasks.rados.rados.0.smithi159.stdout:20: writing smithi15938319-20 from 1962099 to 2319333 tid 2 2024-09-13T13:50:12.342 INFO:tasks.rados.rados.0.smithi159.stdout:21: write initial oid 21 2024-09-13T13:50:12.342 INFO:tasks.rados.rados.0.smithi159.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-13T13:50:12.349 INFO:tasks.rados.rados.0.smithi159.stdout:21: writing smithi15938319-21 from 736140 to 1273826 tid 1 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:21: writing smithi15938319-21 from 2020581 to 2336140 tid 2 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: finishing write tid 2 to smithi15938319-7 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: finishing write tid 3 to smithi15938319-7 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: finishing write tid 4 to smithi15938319-7 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: oid 7 updating version 0 to 7 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: oid 7 updating version 7 to 8 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: oid 7 updating version 8 to 9 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-13T13:50:12.355 INFO:tasks.rados.rados.0.smithi159.stdout:8: finishing write tid 1 to smithi15938319-8 2024-09-13T13:50:12.356 INFO:tasks.rados.rados.0.smithi159.stdout:8: finishing write tid 2 to smithi15938319-8 2024-09-13T13:50:12.356 INFO:tasks.rados.rados.0.smithi159.stdout:7: done (15 left) 2024-09-13T13:50:12.356 INFO:tasks.rados.rados.0.smithi159.stdout:22: write initial oid 22 2024-09-13T13:50:12.356 INFO:tasks.rados.rados.0.smithi159.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-13T13:50:12.362 INFO:tasks.rados.rados.0.smithi159.stdout:22: writing smithi15938319-22 from 752947 to 1365882 tid 1 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:22: writing smithi15938319-22 from 1962710 to 2352947 tid 2 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: finishing write tid 3 to smithi15938319-8 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: finishing write tid 4 to smithi15938319-8 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: oid 8 updating version 0 to 3 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: oid 8 version 3 is already newer than 1 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: oid 8 version 3 is already newer than 2 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-13T13:50:12.368 INFO:tasks.rados.rados.0.smithi159.stdout:9: finishing write tid 1 to smithi15938319-9 2024-09-13T13:50:12.369 INFO:tasks.rados.rados.0.smithi159.stdout:8: done (15 left) 2024-09-13T13:50:12.369 INFO:tasks.rados.rados.0.smithi159.stdout:23: write initial oid 23 2024-09-13T13:50:12.369 INFO:tasks.rados.rados.0.smithi159.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-13T13:50:12.375 INFO:tasks.rados.rados.0.smithi159.stdout:23: writing smithi15938319-23 from 769754 to 1457938 tid 1 2024-09-13T13:50:12.377 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:12.376+0000 7f2282f0c640 -1 osd.10 40 handle_osd_map injecting map crc failure 2024-09-13T13:50:12.380 INFO:tasks.rados.rados.0.smithi159.stdout:23: writing smithi15938319-23 from 2021192 to 2369754 tid 2 2024-09-13T13:50:12.380 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: finishing write tid 2 to smithi15938319-9 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: finishing write tid 3 to smithi15938319-9 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: finishing write tid 4 to smithi15938319-9 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: oid 9 updating version 0 to 1 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: oid 9 updating version 1 to 3 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: oid 9 version 3 is already newer than 2 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:10: finishing write tid 1 to smithi15938319-10 2024-09-13T13:50:12.381 INFO:tasks.rados.rados.0.smithi159.stdout:9: done (15 left) 2024-09-13T13:50:12.382 INFO:tasks.rados.rados.0.smithi159.stdout:24: write initial oid 24 2024-09-13T13:50:12.382 INFO:tasks.rados.rados.0.smithi159.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-13T13:50:12.387 INFO:tasks.rados.rados.0.smithi159.stdout:24: writing smithi15938319-24 from 786561 to 1266347 tid 1 2024-09-13T13:50:12.392 INFO:tasks.rados.rados.0.smithi159.stdout:24: writing smithi15938319-24 from 1796027 to 2263691 tid 2 2024-09-13T13:50:12.392 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.392 INFO:tasks.rados.rados.0.smithi159.stdout:10: finishing write tid 2 to smithi15938319-10 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: finishing write tid 3 to smithi15938319-10 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: finishing write tid 4 to smithi15938319-10 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: oid 10 updating version 0 to 1 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: oid 10 updating version 1 to 3 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: oid 10 version 3 is already newer than 2 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:11: finishing write tid 1 to smithi15938319-11 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:10: done (15 left) 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:25: write initial oid 25 2024-09-13T13:50:12.393 INFO:tasks.rados.rados.0.smithi159.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-13T13:50:12.397 INFO:tasks.rados.rados.0.smithi159.stdout:25: writing smithi15938319-25 from 403368 to 958403 tid 1 2024-09-13T13:50:12.404 INFO:tasks.rados.rados.0.smithi159.stdout:25: writing smithi15938319-25 from 1454509 to 2182184 tid 2 2024-09-13T13:50:12.404 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.404 INFO:tasks.rados.rados.0.smithi159.stdout:11: finishing write tid 2 to smithi15938319-11 2024-09-13T13:50:12.404 INFO:tasks.rados.rados.0.smithi159.stdout:11: finishing write tid 3 to smithi15938319-11 2024-09-13T13:50:12.404 INFO:tasks.rados.rados.0.smithi159.stdout:11: finishing write tid 4 to smithi15938319-11 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:11: oid 11 updating version 0 to 2 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:11: oid 11 updating version 2 to 3 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:11: oid 11 version 3 is already newer than 1 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:12: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:11: done (15 left) 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:26: write initial oid 26 2024-09-13T13:50:12.405 INFO:tasks.rados.rados.0.smithi159.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-13T13:50:12.409 INFO:tasks.rados.rados.0.smithi159.stdout:26: writing smithi15938319-26 from 420175 to 1050459 tid 1 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:26: writing smithi15938319-26 from 1796638 to 2267971 tid 2 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: oid 12 updating version 0 to 3 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: oid 12 version 3 is already newer than 1 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: oid 12 version 3 is already newer than 2 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:5: finishing write tid 3 to smithi15938319-5 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:13: finishing write tid 1 to smithi15938319-13 2024-09-13T13:50:12.418 INFO:tasks.rados.rados.0.smithi159.stdout:13: finishing write tid 2 to smithi15938319-13 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: finishing write tid 3 to smithi15938319-13 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: finishing write tid 4 to smithi15938319-13 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: oid 13 updating version 0 to 1 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: oid 13 updating version 1 to 2 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: oid 13 updating version 2 to 3 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:14: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:14: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: finishing write tid 1 to smithi15938319-15 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: finishing write tid 2 to smithi15938319-15 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: finishing write tid 3 to smithi15938319-15 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: finishing write tid 4 to smithi15938319-15 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: oid 15 updating version 0 to 5 2024-09-13T13:50:12.419 INFO:tasks.rados.rados.0.smithi159.stdout:15: oid 15 version 5 is already newer than 4 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:15: oid 15 updating version 5 to 6 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: finishing write tid 1 to smithi15938319-16 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: finishing write tid 2 to smithi15938319-16 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: finishing write tid 3 to smithi15938319-16 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: finishing write tid 4 to smithi15938319-16 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: oid 16 updating version 0 to 5 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: oid 16 version 5 is already newer than 4 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:16: oid 16 updating version 5 to 6 2024-09-13T13:50:12.420 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:17: finishing write tid 1 to smithi15938319-17 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:17: finishing write tid 2 to smithi15938319-17 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:12: done (15 left) 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:13: done (14 left) 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:15: done (13 left) 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:16: done (12 left) 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:27: write initial oid 27 2024-09-13T13:50:12.421 INFO:tasks.rados.rados.0.smithi159.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-13T13:50:12.423 INFO:tasks.rados.rados.0.smithi159.stdout:27: writing smithi15938319-27 from 436982 to 1142515 tid 1 2024-09-13T13:50:12.423 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:12.422+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:12.429 INFO:tasks.rados.rados.0.smithi159.stdout:27: writing smithi15938319-27 from 1855120 to 2436982 tid 2 2024-09-13T13:50:12.429 INFO:tasks.rados.rados.0.smithi159.stdout:28: write initial oid 28 2024-09-13T13:50:12.430 INFO:tasks.rados.rados.0.smithi159.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-13T13:50:12.436 INFO:tasks.rados.rados.0.smithi159.stdout:28: writing smithi15938319-28 from 453789 to 1234571 tid 1 2024-09-13T13:50:12.442 INFO:tasks.rados.rados.0.smithi159.stdout:28: writing smithi15938319-28 from 1913602 to 2388604 tid 2 2024-09-13T13:50:12.442 INFO:tasks.rados.rados.0.smithi159.stdout:29: write initial oid 29 2024-09-13T13:50:12.442 INFO:tasks.rados.rados.0.smithi159.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-13T13:50:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:12.445 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:12.445 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:12.447 INFO:tasks.rados.rados.0.smithi159.stdout:29: writing smithi15938319-29 from 470596 to 926627 tid 1 2024-09-13T13:50:12.454 INFO:tasks.rados.rados.0.smithi159.stdout:29: writing smithi15938319-29 from 1572084 to 2190744 tid 2 2024-09-13T13:50:12.454 INFO:tasks.rados.rados.0.smithi159.stdout:30: write initial oid 30 2024-09-13T13:50:12.454 INFO:tasks.rados.rados.0.smithi159.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-13T13:50:12.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:12.459 INFO:tasks.rados.rados.0.smithi159.stdout:30: writing smithi15938319-30 from 487403 to 1018683 tid 1 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:30: writing smithi15938319-30 from 1514213 to 1992884 tid 2 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: finishing write tid 3 to smithi15938319-17 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: finishing write tid 4 to smithi15938319-17 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: oid 17 updating version 0 to 9 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: oid 17 version 9 is already newer than 8 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: oid 17 version 9 is already newer than 7 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:12.466 INFO:tasks.rados.rados.0.smithi159.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: finishing write tid 1 to smithi15938319-18 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: finishing write tid 2 to smithi15938319-18 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: finishing write tid 3 to smithi15938319-18 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: finishing write tid 4 to smithi15938319-18 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: oid 18 updating version 0 to 5 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: oid 18 updating version 5 to 6 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: oid 18 version 6 is already newer than 4 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: finishing write tid 1 to smithi15938319-19 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: finishing write tid 2 to smithi15938319-19 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: finishing write tid 3 to smithi15938319-19 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: finishing write tid 4 to smithi15938319-19 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: oid 19 updating version 0 to 6 2024-09-13T13:50:12.467 INFO:tasks.rados.rados.0.smithi159.stdout:19: oid 19 version 6 is already newer than 4 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:19: oid 19 version 6 is already newer than 5 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:17: done (15 left) 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:18: done (14 left) 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:19: done (13 left) 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:31: write initial oid 31 2024-09-13T13:50:12.468 INFO:tasks.rados.rados.0.smithi159.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-13T13:50:12.470 INFO:tasks.rados.rados.0.smithi159.stdout:31: writing smithi15938319-31 from 504210 to 1110739 tid 1 2024-09-13T13:50:12.476 INFO:tasks.rados.rados.0.smithi159.stdout:31: writing smithi15938319-31 from 1572695 to 2195024 tid 2 2024-09-13T13:50:12.476 INFO:tasks.rados.rados.0.smithi159.stdout:32: write initial oid 32 2024-09-13T13:50:12.476 INFO:tasks.rados.rados.0.smithi159.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-13T13:50:12.483 INFO:tasks.rados.rados.0.smithi159.stdout:32: writing smithi15938319-32 from 521017 to 1319148 tid 1 2024-09-13T13:50:12.488 INFO:tasks.rados.rados.0.smithi159.stdout:32: writing smithi15938319-32 from 1747530 to 2229870 tid 2 2024-09-13T13:50:12.488 INFO:tasks.rados.rados.0.smithi159.stdout:33: write initial oid 33 2024-09-13T13:50:12.488 INFO:tasks.rados.rados.0.smithi159.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-13T13:50:12.493 INFO:tasks.rados.rados.0.smithi159.stdout:33: writing smithi15938319-33 from 537824 to 1011204 tid 1 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:33: writing smithi15938319-33 from 1806012 to 2432010 tid 2 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: finishing write tid 1 to smithi15938319-20 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: finishing write tid 2 to smithi15938319-20 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: finishing write tid 3 to smithi15938319-20 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: finishing write tid 4 to smithi15938319-20 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: oid 20 updating version 0 to 3 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: oid 20 version 3 is already newer than 1 2024-09-13T13:50:12.502 INFO:tasks.rados.rados.0.smithi159.stdout:20: oid 20 version 3 is already newer than 2 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:21: finishing write tid 1 to smithi15938319-21 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:20: done (15 left) 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:34: write initial oid 34 2024-09-13T13:50:12.503 INFO:tasks.rados.rados.0.smithi159.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-13T13:50:12.507 INFO:tasks.rados.rados.0.smithi159.stdout:34: writing smithi15938319-34 from 554631 to 1103260 tid 1 2024-09-13T13:50:12.513 INFO:tasks.rados.rados.0.smithi159.stdout:34: writing smithi15938319-34 from 1748141 to 2234150 tid 2 2024-09-13T13:50:12.513 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.513 INFO:tasks.rados.rados.0.smithi159.stdout:21: finishing write tid 2 to smithi15938319-21 2024-09-13T13:50:12.513 INFO:tasks.rados.rados.0.smithi159.stdout:21: finishing write tid 3 to smithi15938319-21 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: finishing write tid 4 to smithi15938319-21 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: oid 21 updating version 0 to 4 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: oid 21 updating version 4 to 6 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: oid 21 version 6 is already newer than 5 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:22: finishing write tid 1 to smithi15938319-22 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:22: finishing write tid 2 to smithi15938319-22 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:22: finishing write tid 3 to smithi15938319-22 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:21: done (15 left) 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:35: write initial oid 35 2024-09-13T13:50:12.514 INFO:tasks.rados.rados.0.smithi159.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-13T13:50:12.519 INFO:tasks.rados.rados.0.smithi159.stdout:35: writing smithi15938319-35 from 571438 to 1195316 tid 1 2024-09-13T13:50:12.524 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:12.523+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:35: writing smithi15938319-35 from 1806623 to 2436290 tid 2 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:22: finishing write tid 4 to smithi15938319-22 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:22: oid 22 updating version 0 to 3 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:22: oid 22 version 3 is already newer than 2 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:22: oid 22 version 3 is already newer than 1 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:23: finishing write tid 1 to smithi15938319-23 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:23: finishing write tid 2 to smithi15938319-23 2024-09-13T13:50:12.527 INFO:tasks.rados.rados.0.smithi159.stdout:23: finishing write tid 3 to smithi15938319-23 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: finishing write tid 4 to smithi15938319-23 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: oid 23 updating version 0 to 3 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: oid 23 version 3 is already newer than 1 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: oid 23 version 3 is already newer than 2 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:22: done (15 left) 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:23: done (14 left) 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:36: write initial oid 36 2024-09-13T13:50:12.528 INFO:tasks.rados.rados.0.smithi159.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-13T13:50:12.533 INFO:tasks.rados.rados.0.smithi159.stdout:36: writing smithi15938319-36 from 588245 to 1287372 tid 1 2024-09-13T13:50:12.538 INFO:tasks.rados.rados.0.smithi159.stdout:36: writing smithi15938319-36 from 1865105 to 2354783 tid 2 2024-09-13T13:50:12.538 INFO:tasks.rados.rados.0.smithi159.stdout:37: write initial oid 37 2024-09-13T13:50:12.538 INFO:tasks.rados.rados.0.smithi159.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-13T13:50:12.546 INFO:tasks.rados.rados.0.smithi159.stdout:37: writing smithi15938319-37 from 605052 to 1379428 tid 1 2024-09-13T13:50:12.552 INFO:tasks.rados.rados.0.smithi159.stdout:37: writing smithi15938319-37 from 1923587 to 2556923 tid 2 2024-09-13T13:50:12.552 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: finishing write tid 1 to smithi15938319-24 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: finishing write tid 2 to smithi15938319-24 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: finishing write tid 3 to smithi15938319-24 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: finishing write tid 4 to smithi15938319-24 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: oid 24 updating version 0 to 9 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: oid 24 version 9 is already newer than 7 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: oid 24 version 9 is already newer than 8 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:25: finishing write tid 1 to smithi15938319-25 2024-09-13T13:50:12.553 INFO:tasks.rados.rados.0.smithi159.stdout:24: done (15 left) 2024-09-13T13:50:12.554 INFO:tasks.rados.rados.0.smithi159.stdout:38: write initial oid 38 2024-09-13T13:50:12.554 INFO:tasks.rados.rados.0.smithi159.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-13T13:50:12.558 INFO:tasks.rados.rados.0.smithi159.stdout:38: writing smithi15938319-38 from 621859 to 1071484 tid 1 2024-09-13T13:50:12.565 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:12.565 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:12.565 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:12.565 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:12.566 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:12.566 INFO:tasks.rados.rados.0.smithi159.stdout:38: writing smithi15938319-38 from 1865716 to 2621859 tid 2 2024-09-13T13:50:12.566 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: finishing write tid 2 to smithi15938319-25 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: finishing write tid 3 to smithi15938319-25 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: finishing write tid 4 to smithi15938319-25 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: oid 25 updating version 0 to 9 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: oid 25 version 9 is already newer than 8 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: oid 25 version 9 is already newer than 7 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:26: finishing write tid 1 to smithi15938319-26 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:25: done (15 left) 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:39: write initial oid 39 2024-09-13T13:50:12.567 INFO:tasks.rados.rados.0.smithi159.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-13T13:50:12.572 INFO:tasks.rados.rados.0.smithi159.stdout:39: writing smithi15938319-39 from 638666 to 1163540 tid 1 2024-09-13T13:50:12.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:39: writing smithi15938319-39 from 1924198 to 2561203 tid 2 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: finishing write tid 2 to smithi15938319-26 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: finishing write tid 3 to smithi15938319-26 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: finishing write tid 4 to smithi15938319-26 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: oid 26 updating version 0 to 4 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: oid 26 updating version 4 to 6 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: oid 26 version 6 is already newer than 5 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-13T13:50:12.580 INFO:tasks.rados.rados.0.smithi159.stdout:27: finishing write tid 1 to smithi15938319-27 2024-09-13T13:50:12.581 INFO:tasks.rados.rados.0.smithi159.stdout:26: done (15 left) 2024-09-13T13:50:12.581 INFO:tasks.rados.rados.0.smithi159.stdout:40: write initial oid 40 2024-09-13T13:50:12.581 INFO:tasks.rados.rados.0.smithi159.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-13T13:50:12.587 INFO:tasks.rados.rados.0.smithi159.stdout:40: writing smithi15938319-40 from 655473 to 1371949 tid 1 2024-09-13T13:50:12.594 INFO:tasks.rados.rados.0.smithi159.stdout:40: writing smithi15938319-40 from 2099033 to 2655473 tid 2 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: finishing write tid 2 to smithi15938319-27 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: finishing write tid 3 to smithi15938319-27 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: finishing write tid 4 to smithi15938319-27 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: oid 27 updating version 0 to 11 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: oid 27 version 11 is already newer than 10 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: oid 27 updating version 11 to 12 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:12.595 INFO:tasks.rados.rados.0.smithi159.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: oid 14 updating version 0 to 5 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: oid 14 version 5 is already newer than 4 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: oid 14 updating version 5 to 6 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:28: finishing write tid 1 to smithi15938319-28 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:28: finishing write tid 2 to smithi15938319-28 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:29: finishing write tid 1 to smithi15938319-29 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:29: finishing write tid 2 to smithi15938319-29 2024-09-13T13:50:12.596 INFO:tasks.rados.rados.0.smithi159.stdout:29: finishing write tid 3 to smithi15938319-29 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: finishing write tid 4 to smithi15938319-29 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: oid 29 updating version 0 to 12 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: oid 29 version 12 is already newer than 10 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: oid 29 version 12 is already newer than 11 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:14: done (15 left) 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:27: done (14 left) 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:29: done (13 left) 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:41: write initial oid 41 2024-09-13T13:50:12.597 INFO:tasks.rados.rados.0.smithi159.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-13T13:50:12.601 INFO:tasks.rados.rados.0.smithi159.stdout:41: writing smithi15938319-41 from 672280 to 1464005 tid 1 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:41: writing smithi15938319-41 from 2157515 to 2672280 tid 2 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:42: write initial oid 42 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: oid 30 updating version 0 to 14 2024-09-13T13:50:12.608 INFO:tasks.rados.rados.0.smithi159.stdout:30: oid 30 updating version 14 to 15 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:30: oid 30 version 15 is already newer than 13 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: finishing write tid 1 to smithi15938319-31 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: finishing write tid 2 to smithi15938319-31 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: finishing write tid 3 to smithi15938319-31 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: finishing write tid 4 to smithi15938319-31 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: oid 31 updating version 0 to 6 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: oid 31 version 6 is already newer than 5 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:31: oid 31 version 6 is already newer than 4 2024-09-13T13:50:12.609 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:12.610 INFO:tasks.rados.rados.0.smithi159.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-13T13:50:12.610 INFO:tasks.rados.rados.0.smithi159.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-13T13:50:12.614 INFO:tasks.rados.rados.0.smithi159.stdout:42: writing smithi15938319-42 from 689087 to 1156061 tid 1 2024-09-13T13:50:12.621 INFO:tasks.rados.rados.0.smithi159.stdout:42: writing smithi15938319-42 from 1699644 to 2483976 tid 2 2024-09-13T13:50:12.621 INFO:tasks.rados.rados.0.smithi159.stdout:30: done (14 left) 2024-09-13T13:50:12.621 INFO:tasks.rados.rados.0.smithi159.stdout:31: done (13 left) 2024-09-13T13:50:12.621 INFO:tasks.rados.rados.0.smithi159.stdout:43: write initial oid 43 2024-09-13T13:50:12.621 INFO:tasks.rados.rados.0.smithi159.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-13T13:50:12.625 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:12.624+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:12.627 INFO:tasks.rados.rados.0.smithi159.stdout:43: writing smithi15938319-43 from 705894 to 1248117 tid 1 2024-09-13T13:50:12.633 INFO:tasks.rados.rados.0.smithi159.stdout:43: writing smithi15938319-43 from 1758126 to 2402469 tid 2 2024-09-13T13:50:12.634 INFO:tasks.rados.rados.0.smithi159.stdout:44: write initial oid 44 2024-09-13T13:50:12.634 INFO:tasks.rados.rados.0.smithi159.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-13T13:50:12.640 INFO:tasks.rados.rados.0.smithi159.stdout:44: writing smithi15938319-44 from 722701 to 1340173 tid 1 2024-09-13T13:50:12.647 INFO:tasks.rados.rados.0.smithi159.stdout:44: writing smithi15938319-44 from 1816608 to 2604609 tid 2 2024-09-13T13:50:12.647 INFO:tasks.rados.rados.0.smithi159.stdout:45: write initial oid 45 2024-09-13T13:50:12.647 INFO:tasks.rados.rados.0.smithi159.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-13T13:50:12.655 INFO:tasks.rados.rados.0.smithi159.stdout:45: writing smithi15938319-45 from 739508 to 1432229 tid 1 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:45: writing smithi15938319-45 from 1875090 to 2523102 tid 2 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: finishing write tid 1 to smithi15938319-32 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: finishing write tid 2 to smithi15938319-32 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: finishing write tid 3 to smithi15938319-32 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: finishing write tid 4 to smithi15938319-32 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: oid 32 updating version 0 to 4 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: oid 32 updating version 4 to 5 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: oid 32 updating version 5 to 6 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-13T13:50:12.662 INFO:tasks.rados.rados.0.smithi159.stdout:33: finishing write tid 1 to smithi15938319-33 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: finishing write tid 2 to smithi15938319-33 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: finishing write tid 3 to smithi15938319-33 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: finishing write tid 4 to smithi15938319-33 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: oid 33 updating version 0 to 5 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: oid 33 updating version 5 to 6 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: oid 33 version 6 is already newer than 4 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:28: finishing write tid 3 to smithi15938319-28 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:32: done (15 left) 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:33: done (14 left) 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:46: write initial oid 46 2024-09-13T13:50:12.663 INFO:tasks.rados.rados.0.smithi159.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-13T13:50:12.669 INFO:tasks.rados.rados.0.smithi159.stdout:46: writing smithi15938319-46 from 756315 to 1524285 tid 1 2024-09-13T13:50:12.676 INFO:tasks.rados.rados.0.smithi159.stdout:46: writing smithi15938319-46 from 1933572 to 2725242 tid 2 2024-09-13T13:50:12.676 INFO:tasks.rados.rados.0.smithi159.stdout:47: write initial oid 47 2024-09-13T13:50:12.676 INFO:tasks.rados.rados.0.smithi159.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-13T13:50:12.683 INFO:tasks.rados.rados.0.smithi159.stdout:47: writing smithi15938319-47 from 773122 to 1332694 tid 1 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:9 4.b e41/34 4:def5de2b:::smithi15938319-3:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.145496+0000", 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.53954147900000005, 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045996630000000004, 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.687 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.145496+0000", 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.145496+0000", 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.145496+0000", 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146318+0000", 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00082159900000000001 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146365+0000", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7550999999999999e-05 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146379+0000", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3657e-05 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146448+0000", 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9189000000000005e-05 2024-09-13T13:50:12.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146472+0000", 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.41e-05 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150049+0000", 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035768250000000001 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150095+0000", 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6276999999999998e-05 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:10 4.b e41/34 4:def5de2b:::smithi15938319-3:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.146820+0000", 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.53821677599999995, 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033177110000000001, 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.692 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146820+0000", 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146820+0000", 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146821+0000", 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.23e-07 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.147123+0000", 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000302102 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.147158+0000", 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5197000000000002e-05 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.147166+0000", 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0679999999999999e-06 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.147591+0000", 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042526599999999999 2024-09-13T13:50:12.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.147625+0000", 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150105+0000", 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002480337 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150138+0000", 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2629999999999998e-05 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:25 4.b e41/34 4:d7a30b61:::smithi15938319-7:head v 41'7, mlcod=41'5)", 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.181157+0000", 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.50388060599999995, 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034753330000000002, 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.697 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181157+0000", 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181157+0000", 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181157+0000", 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7300000000000005e-07 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181411+0000", 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00025330200000000001 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181452+0000", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1023999999999999e-05 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181466+0000", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4037e-05 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181519+0000", 2024-09-13T13:50:12.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.384e-05 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.181535+0000", 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5460000000000001e-05 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.184584+0000", 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030488899999999998 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.184632+0000", 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8007000000000003e-05 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:37 4.1 4:8e4bdd2a:::smithi15938319-10:head [write 551263~793594,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.222209+0000", 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.46282854099999998, 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036088209999999999, 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.702 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 37 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.222209+0000", 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.222209+0000", 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.222210+0000", 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.291e-06 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223043+0000", 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00083311300000000002 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223054+0000", 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0811e-05 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223060+0000", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9599999999999997e-06 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223121+0000", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1111999999999994e-05 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223218+0000", 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7254000000000001e-05 2024-09-13T13:50:12.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223320+0000", 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101489 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225441+0000", 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002121146 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225738+0000", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00029723100000000001 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225772+0000", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3868000000000002e-05 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225817+0000", 2024-09-13T13:50:12.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5546000000000001e-05 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:39 4.1 4:8e4bdd2a:::smithi15938319-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.223277+0000", 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.461760214, 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045830009999999997, 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.709 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 39 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223277+0000", 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223277+0000", 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223277+0000", 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223282+0000", 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0600000000000001e-06 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223287+0000", 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9479999999999996e-06 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223293+0000", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6200000000000004e-06 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225086+0000", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017926210000000001 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225134+0000", 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8038000000000003e-05 2024-09-13T13:50:12.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225236+0000", 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102503 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.226106+0000", 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087023600000000003 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.227780+0000", 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001673522 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.227813+0000", 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3429000000000001e-05 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.227860+0000", 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6471000000000003e-05 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:40 4.1 4:8e4bdd2a:::smithi15938319-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.223299+0000", 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.46173793899999999, 2024-09-13T13:50:12.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047077919999999997, 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 40 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223299+0000", 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223299+0000", 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223300+0000", 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223302+0000", 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2730000000000001e-06 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223307+0000", 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8230000000000002e-06 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.223310+0000", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0850000000000001e-06 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225484+0000", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021748010000000001 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.225530+0000", 2024-09-13T13:50:12.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5710000000000001e-05 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.227895+0000", 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023644640000000001 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.227938+0000", 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3600999999999998e-05 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.228007+0000", 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8707000000000005e-05 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:42 4.f e41/34 4:fec51f4e:::smithi15938319-11:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.239270+0000", 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.44576733200000002, 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038968050000000001, 2024-09-13T13:50:12.722 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239270+0000", 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239270+0000", 2024-09-13T13:50:12.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239271+0000", 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8100000000000002e-07 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239810+0000", 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053965299999999996 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239839+0000", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8405e-05 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239852+0000", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239899+0000", 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7382999999999997e-05 2024-09-13T13:50:12.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.239914+0000", 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4528999999999999e-05 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.243123+0000", 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032092879999999998 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.243167+0000", 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3339e-05 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:43 4.f e41/34 4:fec51f4e:::smithi15938319-11:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.240068+0000", 2024-09-13T13:50:12.727 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.44496928099999999, 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003319055, 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240068+0000", 2024-09-13T13:50:12.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240068+0000", 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240068+0000", 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9599999999999999e-07 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240074+0000", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9649999999999996e-06 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240095+0000", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0531e-05 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240100+0000", 2024-09-13T13:50:12.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6290000000000001e-06 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240512+0000", 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041216499999999998 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.240533+0000", 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0905000000000001e-05 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.243352+0000", 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028197080000000002 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.243387+0000", 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4555999999999999e-05 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.732 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:65 4.1 4:8e40678c:::smithi15938319-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.300526+0000", 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.38451112700000001, 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004777, 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 65 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.300526+0000", 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.300526+0000", 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.300528+0000", 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0729999999999999e-06 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301054+0000", 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052552100000000004 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301065+0000", 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1157e-05 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301070+0000", 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.429e-06 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301133+0000", 2024-09-13T13:50:12.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3065999999999996e-05 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301222+0000", 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9061000000000007e-05 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.301303+0000", 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0258000000000003e-05 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.302297+0000", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00099453600000000007 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.305232+0000", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029344179999999998 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.305264+0000", 2024-09-13T13:50:12.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.222e-05 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.305303+0000", 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9261000000000002e-05 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:94 4.e e41/34 4:717ade62:::smithi15938319-24:head v 41'8, mlcod=41'6)", 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.393511+0000", 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.29152650099999999, 2024-09-13T13:50:12.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0037420349999999999, 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393511+0000", 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393511+0000", 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393511+0000", 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.61e-07 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393891+0000", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037991799999999998 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393914+0000", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2861999999999999e-05 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393920+0000", 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8710000000000002e-06 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393952+0000", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1610999999999999e-05 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393963+0000", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1657e-05 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.397223+0000", 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032598390000000001 2024-09-13T13:50:12.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.397253+0000", 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9615999999999999e-05 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:95 4.e e41/34 4:717ade62:::smithi15938319-24:head v 41'9, mlcod=41'6)", 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.393987+0000", 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.29105007900000002, 2024-09-13T13:50:12.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032878830000000001, 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393987+0000", 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393987+0000", 2024-09-13T13:50:12.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393987+0000", 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0499999999999999e-07 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.393991+0000", 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3139999999999999e-06 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.394004+0000", 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3678e-05 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.394008+0000", 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.129e-06 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.394276+0000", 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00026866299999999999 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.394290+0000", 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3434999999999999e-05 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.397262+0000", 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029720929999999999 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.397275+0000", 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3166000000000001e-05 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:114 4.1 4:81ab2613:::smithi15938319-29:head [write 1572084~618660] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.454318+0000", 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.23071872500000001, 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049720399999999996, 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 114 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.454318+0000", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.454318+0000", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.454324+0000", 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2920000000000003e-06 2024-09-13T13:50:12.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455411+0000", 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001087544 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455427+0000", 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5730999999999999e-05 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455434+0000", 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3509999999999996e-06 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455484+0000", 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0138e-05 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455547+0000", 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2538000000000003e-05 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455636+0000", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8727999999999995e-05 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.457347+0000", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017113009999999999 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459227+0000", 2024-09-13T13:50:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001880163 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459261+0000", 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3793e-05 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459290+0000", 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9461000000000001e-05 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:115 4.1 4:81ab2613:::smithi15938319-29:head [setxattr _header (56) in=7b,truncate 2470596] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.455452+0000", 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.22958497, 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040781380000000002, 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 115 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455452+0000", 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455452+0000", 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455453+0000", 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3300000000000003e-07 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455456+0000", 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2969999999999999e-06 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455460+0000", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5299999999999998e-06 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455465+0000", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3259999999999997e-06 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.456528+0000", 2024-09-13T13:50:12.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00106309 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.456581+0000", 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3303000000000001e-05 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.456657+0000", 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5926000000000005e-05 2024-09-13T13:50:12.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.457862+0000", 2024-09-13T13:50:12.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00120499 2024-09-13T13:50:12.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459477+0000", 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016153179999999999 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459500+0000", 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2966e-05 2024-09-13T13:50:12.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459530+0000", 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9958999999999999e-05 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:116 4.1 4:81ab2613:::smithi15938319-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.455476+0000", 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.229561021, 2024-09-13T13:50:12.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041860320000000001, 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.764 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455476+0000", 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455476+0000", 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455476+0000", 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455478+0000", 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5349999999999999e-06 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455482+0000", 2024-09-13T13:50:12.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0779999999999997e-06 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.455485+0000", 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0900000000000001e-06 2024-09-13T13:50:12.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.456819+0000", 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001334121 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:12.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.456878+0000", 2024-09-13T13:50:12.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8440999999999998e-05 2024-09-13T13:50:12.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459558+0000", 2024-09-13T13:50:12.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026803719999999999 2024-09-13T13:50:12.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459601+0000", 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3349000000000001e-05 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.459662+0000", 2024-09-13T13:50:12.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0717999999999997e-05 2024-09-13T13:50:12.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.774 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.775 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:118 4.b e41/34 4:d69faa47:::smithi15938319-30:head v 41'14, mlcod=41'12)", 2024-09-13T13:50:12.775 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.465688+0000", 2024-09-13T13:50:12.775 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.219349345, 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035182830000000001, 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465688+0000", 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465688+0000", 2024-09-13T13:50:12.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465688+0000", 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465788+0000", 2024-09-13T13:50:12.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9193999999999999e-05 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465813+0000", 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5219e-05 2024-09-13T13:50:12.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465819+0000", 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0000000000000002e-06 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465872+0000", 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2902999999999998e-05 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.465895+0000", 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3107000000000001e-05 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.469160+0000", 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032647399999999999 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.469206+0000", 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6468000000000003e-05 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:137 4.1 4:80c58c2d:::smithi15938319-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.783 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.519790+0000", 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.165247318, 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052539939999999997, 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.519790+0000", 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.519790+0000", 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.519792+0000", 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3360000000000002e-06 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520387+0000", 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059499499999999996 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520404+0000", 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7139e-05 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520412+0000", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8820000000000001e-06 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520475+0000", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2957e-05 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520610+0000", 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013505799999999999 2024-09-13T13:50:12.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.520761+0000", 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015081900000000001 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.523988+0000", 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032268470000000001 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.524973+0000", 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00098513099999999998 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.525009+0000", 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6165999999999999e-05 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.525044+0000", 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4663999999999999e-05 2024-09-13T13:50:12.789 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:138 4.1 4:80c58c2d:::smithi15938319-35:head [write 1806623~629667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.526332+0000", 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.15870473199999999, 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005252339, 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.790 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 138 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.526332+0000", 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.526332+0000", 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.526333+0000", 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527116+0000", 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000782542 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527128+0000", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2038999999999999e-05 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527134+0000", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4080000000000003e-06 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527176+0000", 2024-09-13T13:50:12.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1621e-05 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527229+0000", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3331000000000002e-05 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527315+0000", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5446999999999997e-05 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.528424+0000", 2024-09-13T13:50:12.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001109016 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531505+0000", 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030812639999999998 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531542+0000", 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6545000000000003e-05 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531585+0000", 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3167000000000002e-05 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:139 4.1 4:80c58c2d:::smithi15938319-35:head [setxattr _header (56) in=7b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.527146+0000", 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.15789123499999999, 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045442210000000002, 2024-09-13T13:50:12.796 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527146+0000", 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527146+0000", 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527146+0000", 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1399999999999997e-07 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527148+0000", 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0389999999999999e-06 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527152+0000", 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527155+0000", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.853e-06 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527926+0000", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00077117799999999997 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527979+0000", 2024-09-13T13:50:12.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3482000000000001e-05 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.528070+0000", 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0587000000000005e-05 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.530667+0000", 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025973210000000001 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531617+0000", 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094972299999999997 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531647+0000", 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0249000000000001e-05 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531690+0000", 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3035999999999999e-05 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:140 4.1 4:80c58c2d:::smithi15938319-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.527171+0000", 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.15786614400000001, 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047224069999999996, 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.803 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 140 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527171+0000", 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527171+0000", 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527171+0000", 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527173+0000", 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.463e-06 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527176+0000", 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6500000000000002e-06 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.527180+0000", 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2430000000000001e-06 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.528232+0000", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010520270000000001 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.528272+0000", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9876000000000002e-05 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531724+0000", 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003452335 2024-09-13T13:50:12.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531794+0000", 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0355e-05 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.531893+0000", 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9129999999999994e-05 2024-09-13T13:50:12.808 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:158 4.f e41/34 4:f883ee18:::smithi15938319-40:head v 41'11, mlcod=41'9)", 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.594647+0000", 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.090390543000000004, 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035674119999999998, 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.594647+0000", 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.594647+0000", 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.594647+0000", 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8299999999999995e-07 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.595103+0000", 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045508400000000002 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.595127+0000", 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4164000000000001e-05 2024-09-13T13:50:12.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.595135+0000", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0690000000000002e-06 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.595187+0000", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2658999999999998e-05 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.595208+0000", 2024-09-13T13:50:12.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1019999999999999e-05 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.598180+0000", 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029717350000000001 2024-09-13T13:50:12.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:12.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.598214+0000", 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3797999999999997e-05 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:12.819 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:12.817+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:12.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:12.839 INFO:tasks.rados.rados.0.smithi159.stdout:47: writing smithi15938319-47 from 1992054 to 2643735 tid 2 2024-09-13T13:50:12.839 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:28: finishing write tid 4 to smithi15938319-28 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:28: oid 28 updating version 0 to 8 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:28: oid 28 updating version 8 to 9 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:28: oid 28 version 9 is already newer than 7 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:5: finishing write tid 4 to smithi15938319-5 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:5: oid 5 updating version 0 to 2 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:5: oid 5 updating version 2 to 3 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:5: oid 5 version 3 is already newer than 1 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:12.840 INFO:tasks.rados.rados.0.smithi159.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:34: finishing write tid 1 to smithi15938319-34 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:35: finishing write tid 1 to smithi15938319-35 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:5: done (15 left) 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:28: done (14 left) 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:48: write initial oid 48 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:48: writing smithi15938319-48 from 789929 to 1424750 tid 1 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:48: writing smithi15938319-48 from 2050536 to 2789929 tid 2 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:49: write initial oid 49 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:49: writing smithi15938319-49 from 406736 to 1116806 tid 1 2024-09-13T13:50:12.841 INFO:tasks.rados.rados.0.smithi159.stdout:49: writing smithi15938319-49 from 1709018 to 2364368 tid 2 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: finishing write tid 2 to smithi15938319-35 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: finishing write tid 3 to smithi15938319-35 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: finishing write tid 4 to smithi15938319-35 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: oid 35 updating version 0 to 14 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: oid 35 updating version 14 to 15 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: oid 35 version 15 is already newer than 13 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:36: finishing write tid 1 to smithi15938319-36 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:36: finishing write tid 2 to smithi15938319-36 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:36: finishing write tid 3 to smithi15938319-36 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:36: finishing write tid 4 to smithi15938319-36 2024-09-13T13:50:12.842 INFO:tasks.rados.rados.0.smithi159.stdout:36: oid 36 updating version 0 to 9 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:36: oid 36 version 9 is already newer than 7 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:36: oid 36 version 9 is already newer than 8 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:34: finishing write tid 2 to smithi15938319-34 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:34: finishing write tid 3 to smithi15938319-34 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:35: done (15 left) 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:36: done (14 left) 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:50: write initial oid 50 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:50: writing smithi15938319-50 from 423543 to 1208862 tid 1 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:50: writing smithi15938319-50 from 1767500 to 2566508 tid 2 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:51: copy oid 2 target oid 2 low_tier 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.843 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: finishing write tid 1 to smithi15938319-37 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: finishing write tid 2 to smithi15938319-37 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: finishing write tid 3 to smithi15938319-37 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: finishing write tid 4 to smithi15938319-37 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: oid 37 updating version 0 to 12 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: oid 37 version 12 is already newer than 11 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: oid 37 version 12 is already newer than 10 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:34: finishing write tid 4 to smithi15938319-34 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:34: oid 34 updating version 0 to 4 2024-09-13T13:50:12.844 INFO:tasks.rados.rados.0.smithi159.stdout:34: oid 34 updating version 4 to 5 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:34: oid 34 updating version 5 to 6 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:38: finishing write tid 1 to smithi15938319-38 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: finishing write tid 1 to smithi15938319-39 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: finishing write tid 2 to smithi15938319-39 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: finishing write tid 3 to smithi15938319-39 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: finishing write tid 4 to smithi15938319-39 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: oid 39 updating version 0 to 7 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: oid 39 updating version 7 to 9 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: oid 39 version 9 is already newer than 8 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:12.845 INFO:tasks.rados.rados.0.smithi159.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: finishing write tid 1 to smithi15938319-40 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: finishing write tid 2 to smithi15938319-40 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: finishing write tid 3 to smithi15938319-40 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: finishing write tid 4 to smithi15938319-40 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: oid 40 updating version 0 to 12 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: oid 40 version 12 is already newer than 10 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: oid 40 version 12 is already newer than 11 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:41: finishing write tid 1 to smithi15938319-41 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:41: finishing write tid 2 to smithi15938319-41 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:41: finishing write tid 3 to smithi15938319-41 2024-09-13T13:50:12.846 INFO:tasks.rados.rados.0.smithi159.stdout:41: finishing write tid 4 to smithi15938319-41 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:41: oid 41 updating version 0 to 8 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:41: oid 41 updating version 8 to 9 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:41: oid 41 version 9 is already newer than 7 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: finishing write tid 1 to smithi15938319-42 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: finishing write tid 2 to smithi15938319-42 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: finishing write tid 3 to smithi15938319-42 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: finishing write tid 4 to smithi15938319-42 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: oid 42 updating version 0 to 8 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: oid 42 version 8 is already newer than 7 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: oid 42 updating version 8 to 9 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:12.847 INFO:tasks.rados.rados.0.smithi159.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: finishing write tid 1 to smithi15938319-43 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: finishing write tid 2 to smithi15938319-43 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: finishing write tid 3 to smithi15938319-43 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: finishing write tid 4 to smithi15938319-43 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: oid 43 updating version 0 to 11 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: oid 43 updating version 11 to 12 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: oid 43 version 12 is already newer than 10 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:44: finishing write tid 1 to smithi15938319-44 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:44: finishing write tid 2 to smithi15938319-44 2024-09-13T13:50:12.848 INFO:tasks.rados.rados.0.smithi159.stdout:44: finishing write tid 3 to smithi15938319-44 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:44: finishing write tid 4 to smithi15938319-44 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:44: oid 44 updating version 0 to 12 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:44: oid 44 version 12 is already newer than 10 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:44: oid 44 version 12 is already newer than 11 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:45: finishing write tid 1 to smithi15938319-45 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:34: done (15 left) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:37: done (14 left) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:39: done (13 left) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:40: done (12 left) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:41: done (11 left) 2024-09-13T13:50:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:42: done (10 left) 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:43: done (9 left) 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:44: done (8 left) 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:53: copy oid 4 target oid 4 low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:55: copy oid 6 target oid 6 low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.850 INFO:tasks.rados.rados.0.smithi159.stdout:57: copy oid 8 target oid 8 low_tier 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:59: copy oid 10 target oid 10 low_tier 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: finishing write tid 2 to smithi15938319-45 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: finishing write tid 3 to smithi15938319-45 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: finishing write tid 4 to smithi15938319-45 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: oid 45 updating version 0 to 1 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: oid 45 updating version 1 to 3 2024-09-13T13:50:12.851 INFO:tasks.rados.rados.0.smithi159.stdout:45: oid 45 version 3 is already newer than 2 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 3 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:46: finishing write tid 1 to smithi15938319-46 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:45: done (15 left) 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:46: finishing write tid 2 to smithi15938319-46 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:46: finishing write tid 3 to smithi15938319-46 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:46: finishing write tid 4 to smithi15938319-46 2024-09-13T13:50:12.852 INFO:tasks.rados.rados.0.smithi159.stdout:46: oid 46 updating version 0 to 8 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:46: oid 46 updating version 8 to 9 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:46: oid 46 version 9 is already newer than 7 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:46: done (15 left) 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:61: copy oid 12 target oid 12 low_tier 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:38: finishing write tid 2 to smithi15938319-38 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:38: finishing write tid 3 to smithi15938319-38 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:38: finishing write tid 4 to smithi15938319-38 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:38: oid 38 updating version 0 to 7 2024-09-13T13:50:12.853 INFO:tasks.rados.rados.0.smithi159.stdout:38: oid 38 updating version 7 to 8 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:38: oid 38 updating version 8 to 9 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:47: finishing write tid 1 to smithi15938319-47 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:38: done (15 left) 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:48: finishing write tid 1 to smithi15938319-48 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:48: finishing write tid 2 to smithi15938319-48 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:48: finishing write tid 3 to smithi15938319-48 2024-09-13T13:50:12.854 INFO:tasks.rados.rados.0.smithi159.stdout:48: finishing write tid 4 to smithi15938319-48 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:48: oid 48 updating version 0 to 18 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:48: oid 48 version 18 is already newer than 16 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:48: oid 48 version 18 is already newer than 17 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:48: done (15 left) 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:63: copy oid 14 target oid 14 low_tier 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:49: finishing write tid 1 to smithi15938319-49 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:49: finishing write tid 2 to smithi15938319-49 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:49: finishing write tid 3 to smithi15938319-49 2024-09-13T13:50:12.855 INFO:tasks.rados.rados.0.smithi159.stdout:49: finishing write tid 4 to smithi15938319-49 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:49: oid 49 updating version 0 to 2 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:49: oid 49 version 2 is already newer than 1 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:49: oid 49 updating version 2 to 3 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 3 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:50: finishing write tid 1 to smithi15938319-50 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:49: done (15 left) 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:50: finishing write tid 2 to smithi15938319-50 2024-09-13T13:50:12.856 INFO:tasks.rados.rados.0.smithi159.stdout:50: finishing write tid 3 to smithi15938319-50 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: finishing write tid 4 to smithi15938319-50 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: oid 50 updating version 0 to 12 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: oid 50 version 12 is already newer than 11 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: oid 50 version 12 is already newer than 10 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:50: done (15 left) 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:65: copy oid 16 target oid 16 low_tier 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:62: done (15 left) 2024-09-13T13:50:12.857 INFO:tasks.rados.rados.0.smithi159.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:51: finishing copy op to oid 2 low_tier 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:51: done (15 left) 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:67: copy oid 18 target oid 18 low_tier 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:63: finishing copy op to oid 14 low_tier 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:63: done (15 left) 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.858 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:54: done (15 left) 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:69: copy oid 20 target oid 20 low_tier 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:58: done (15 left) 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.859 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:56: done (15 left) 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:71: copy oid 22 target oid 22 low_tier 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:57: finishing copy op to oid 8 low_tier 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:57: done (15 left) 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.860 INFO:tasks.rados.rados.0.smithi159.stdout:64: done (15 left) 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:73: copy oid 24 target oid 24 low_tier 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:60: done (15 left) 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.861 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:55: finishing copy op to oid 6 low_tier 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:55: done (15 left) 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:75: copy oid 26 target oid 26 low_tier 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:59: finishing copy op to oid 10 low_tier 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:59: done (15 left) 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:52: done (15 left) 2024-09-13T13:50:12.862 INFO:tasks.rados.rados.0.smithi159.stdout:77: copy oid 28 target oid 28 low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:61: finishing copy op to oid 12 low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:61: done (15 left) 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:65: finishing copy op to oid 16 low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:65: done (15 left) 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:79: copy oid 30 target oid 30 low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.863 INFO:tasks.rados.rados.0.smithi159.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:68: done (15 left) 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:66: done (15 left) 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:81: copy oid 32 target oid 32 low_tier 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:74: done (15 left) 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.864 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:53: finishing copy op to oid 4 low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:53: done (15 left) 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:83: copy oid 34 target oid 34 low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:67: finishing copy op to oid 18 low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:67: done (15 left) 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.865 INFO:tasks.rados.rados.0.smithi159.stdout:73: finishing copy op to oid 24 low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:73: done (15 left) 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:85: copy oid 36 target oid 36 low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:75: finishing copy op to oid 26 low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:75: done (15 left) 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:69: finishing copy op to oid 20 low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:69: done (15 left) 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:87: copy oid 38 target oid 38 low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.866 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:78: done (15 left) 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:71: finishing copy op to oid 22 low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:71: done (15 left) 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:89: copy oid 40 target oid 40 low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:77: finishing copy op to oid 28 low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:77: done (15 left) 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.867 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:76: done (15 left) 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:91: copy oid 42 target oid 42 low_tier 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:72: done (15 left) 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:79: finishing copy op to oid 30 low_tier 2024-09-13T13:50:12.868 INFO:tasks.rados.rados.0.smithi159.stdout:79: done (15 left) 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:93: copy oid 44 target oid 44 low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:83: finishing copy op to oid 34 low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:83: done (15 left) 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:84: done (15 left) 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:95: copy oid 46 target oid 46 low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.869 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:70: done (15 left) 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:95 wait for completion of write op! 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:80: done (14 left) 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:81: finishing copy op to oid 32 low_tier 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:81: done (13 left) 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:82: done (12 left) 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:85: finishing copy op to oid 36 low_tier 2024-09-13T13:50:12.870 INFO:tasks.rados.rados.0.smithi159.stdout:85: done (11 left) 2024-09-13T13:50:12.871 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:12.871 INFO:tasks.rados.rados.0.smithi159.stdout:91: finishing copy op to oid 42 low_tier 2024-09-13T13:50:12.871 INFO:tasks.rados.rados.0.smithi159.stdout:91: done (10 left) 2024-09-13T13:50:12.871 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:92: done (9 left) 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:94: done (8 left) 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:86: done (7 left) 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:87: finishing copy op to oid 38 low_tier 2024-09-13T13:50:12.873 INFO:tasks.rados.rados.0.smithi159.stdout:87: done (6 left) 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:88: done (5 left) 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:90: done (4 left) 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:95: finishing copy op to oid 46 low_tier 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:95: done (3 left) 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:93: finishing copy op to oid 44 low_tier 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:93: done (2 left) 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:89: finishing copy op to oid 40 low_tier 2024-09-13T13:50:12.874 INFO:tasks.rados.rados.0.smithi159.stdout:89: done (1 left) 2024-09-13T13:50:12.875 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:12.899 INFO:tasks.rados.rados.0.smithi159.stdout:47: finishing write tid 2 to smithi15938319-47 2024-09-13T13:50:12.899 INFO:tasks.rados.rados.0.smithi159.stdout:47: finishing write tid 3 to smithi15938319-47 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: finishing write tid 4 to smithi15938319-47 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: oid 47 updating version 0 to 10 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: oid 47 updating version 10 to 11 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: oid 47 updating version 11 to 12 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:47: done (0 left) 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:97: copy oid 48 target oid 48 low_tier 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.900 INFO:tasks.rados.rados.0.smithi159.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:99: copy oid 50 target oid 50 low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:101: set_chunk oid 2 offset: 1188352 length: 115200 target oid smithi15938319-2 low_tier offset: 1188352 snap -1 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:102: set_chunk oid 3 offset: 544768 length: 756736 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 544768 snap -1 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:12.901 INFO:tasks.rados.rados.0.smithi159.stdout:103: set_chunk oid 4 offset: 245760 length: 1406464 target oid smithi15938319-4 low_tier offset: 245760 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:104: set_chunk oid 5 offset: 955392 length: 482304 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 955392 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:105: set_chunk oid 6 offset: 865792 length: 1124352 target oid smithi15938319-6 low_tier offset: 865792 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:106: set_chunk oid 7 offset: 271360 length: 1057280 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 271360 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:107: set_chunk oid 8 offset: 747520 length: 559616 target oid smithi15938319-8 low_tier offset: 747520 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:108: set_chunk oid 9 offset: 28672 length: 837120 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 28672 snap -1 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:12.902 INFO:tasks.rados.rados.0.smithi159.stdout:109: set_chunk oid 10 offset: 929792 length: 1184768 target oid smithi15938319-10 low_tier offset: 929792 snap -1 2024-09-13T13:50:12.903 INFO:tasks.rados.rados.0.smithi159.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.903 INFO:tasks.rados.rados.0.smithi159.stdout:110: set_chunk oid 11 offset: 1406464 length: 750080 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1406464 snap -1 2024-09-13T13:50:12.903 INFO:tasks.rados.rados.0.smithi159.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:12.903 INFO:tasks.rados.rados.0.smithi159.stdout:111: set_chunk oid 12 offset: 925184 length: 437760 target oid smithi15938319-12 low_tier offset: 925184 snap -1 2024-09-13T13:50:12.903 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:109: finishing set_chunk to oid 10 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:109: done (15 left) 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:112: set_chunk oid 13 offset: 1869824 length: 288256 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1869824 snap -1 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:108: finishing set_chunk to oid 9 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:108: done (15 left) 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:12.910 INFO:tasks.rados.rados.0.smithi159.stdout:113: set_chunk oid 14 offset: 48128 length: 704000 target oid smithi15938319-14 low_tier offset: 48128 snap -1 2024-09-13T13:50:12.911 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.911 INFO:tasks.rados.rados.0.smithi159.stdout:111: finishing set_chunk to oid 12 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:111: done (15 left) 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:114: set_chunk oid 15 offset: 380416 length: 1033216 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 380416 snap -1 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:97: finishing copy op to oid 48 low_tier 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:97: done (15 left) 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout:115: set_chunk oid 16 offset: 1552896 length: 612864 target oid smithi15938319-16 low_tier offset: 1552896 snap -1 2024-09-13T13:50:12.912 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.913 INFO:tasks.rados.rados.0.smithi159.stdout:107: finishing set_chunk to oid 8 2024-09-13T13:50:12.914 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:12.914 INFO:tasks.rados.rados.0.smithi159.stdout:107: done (15 left) 2024-09-13T13:50:12.914 INFO:tasks.rados.rados.0.smithi159.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.914 INFO:tasks.rados.rados.0.smithi159.stdout:116: set_chunk oid 17 offset: 76800 length: 596992 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 76800 snap -1 2024-09-13T13:50:12.914 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:105: finishing set_chunk to oid 6 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:105: done (15 left) 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:117: set_chunk oid 18 offset: 952320 length: 83968 target oid smithi15938319-18 low_tier offset: 952320 snap -1 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:106: finishing set_chunk to oid 7 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:12.916 INFO:tasks.rados.rados.0.smithi159.stdout:106: done (15 left) 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:118: set_chunk oid 19 offset: 506880 length: 829952 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 506880 snap -1 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:110: finishing set_chunk to oid 11 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:110: done (15 left) 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:119: set_chunk oid 20 offset: 1383936 length: 438272 target oid smithi15938319-20 low_tier offset: 1383936 snap -1 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:102: finishing set_chunk to oid 3 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:102: done (15 left) 2024-09-13T13:50:12.917 INFO:tasks.rados.rados.0.smithi159.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.918 INFO:tasks.rados.rados.0.smithi159.stdout:120: set_chunk oid 21 offset: 222720 length: 1859584 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 222720 snap -1 2024-09-13T13:50:12.918 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:112: finishing set_chunk to oid 13 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:112: done (15 left) 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:121: set_chunk oid 22 offset: 1569792 length: 558592 target oid smithi15938319-22 low_tier offset: 1569792 snap -1 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:100: done (15 left) 2024-09-13T13:50:12.919 INFO:tasks.rados.rados.0.smithi159.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.920 INFO:tasks.rados.rados.0.smithi159.stdout:122: set_chunk oid 23 offset: 415232 length: 498688 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 415232 snap -1 2024-09-13T13:50:12.920 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.920 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:12.918+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:115: finishing set_chunk to oid 16 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:115: done (15 left) 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:123: set_chunk oid 24 offset: 1838080 length: 77312 target oid smithi15938319-24 low_tier offset: 1838080 snap -1 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.923 INFO:tasks.rados.rados.0.smithi159.stdout:101: finishing set_chunk to oid 2 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:101: done (15 left) 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:124: set_chunk oid 25 offset: 646656 length: 954368 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 646656 snap -1 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:98: done (15 left) 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:125: set_chunk oid 26 offset: 132608 length: 722944 target oid smithi15938319-26 low_tier offset: 132608 snap -1 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.924 INFO:tasks.rados.rados.0.smithi159.stdout:103: finishing set_chunk to oid 4 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:103: done (15 left) 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:126: set_chunk oid 27 offset: 1788928 length: 548864 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1788928 snap -1 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:114: finishing set_chunk to oid 15 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:114: done (15 left) 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:127: set_chunk oid 28 offset: 1974272 length: 446976 target oid smithi15938319-28 low_tier offset: 1974272 snap -1 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:117: finishing set_chunk to oid 18 2024-09-13T13:50:12.925 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:117: done (15 left) 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:128: set_chunk oid 29 offset: 122880 length: 138240 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 122880 snap -1 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:123: finishing set_chunk to oid 24 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:123: done (15 left) 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout:129: set_chunk oid 30 offset: 10752 length: 925184 target oid smithi15938319-30 low_tier offset: 10752 snap -1 2024-09-13T13:50:12.926 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.928 INFO:tasks.rados.rados.0.smithi159.stdout:99: finishing copy op to oid 50 low_tier 2024-09-13T13:50:12.928 INFO:tasks.rados.rados.0.smithi159.stdout:99: done (15 left) 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:130: set_chunk oid 31 offset: 452608 length: 1590272 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 452608 snap -1 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:116: finishing set_chunk to oid 17 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:116: done (15 left) 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:131: set_chunk oid 32 offset: 635392 length: 857600 target oid smithi15938319-32 low_tier offset: 635392 snap -1 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:127: finishing set_chunk to oid 28 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:127: done (15 left) 2024-09-13T13:50:12.929 INFO:tasks.rados.rados.0.smithi159.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:132: set_chunk oid 33 offset: 222720 length: 2293760 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 222720 snap -1 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:118: finishing set_chunk to oid 19 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:118: done (15 left) 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout:133: set_chunk oid 34 offset: 1248768 length: 708096 target oid smithi15938319-34 low_tier offset: 1248768 snap -1 2024-09-13T13:50:12.930 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.933 INFO:tasks.rados.rados.0.smithi159.stdout:126: finishing set_chunk to oid 27 2024-09-13T13:50:12.933 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:12.933 INFO:tasks.rados.rados.0.smithi159.stdout:126: done (15 left) 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:134: set_chunk oid 35 offset: 1724928 length: 9216 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1724928 snap -1 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:122: finishing set_chunk to oid 23 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:122: done (15 left) 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:135: set_chunk oid 36 offset: 1197568 length: 703488 target oid smithi15938319-36 low_tier offset: 1197568 snap -1 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:113: finishing set_chunk to oid 14 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:12.934 INFO:tasks.rados.rados.0.smithi159.stdout:113: done (15 left) 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:136: set_chunk oid 37 offset: 541184 length: 993280 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 541184 snap -1 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:130: finishing set_chunk to oid 31 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:130: done (15 left) 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:137: set_chunk oid 38 offset: 2282496 length: 245248 target oid smithi15938319-38 low_tier offset: 2282496 snap -1 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:121: finishing set_chunk to oid 22 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:121: done (15 left) 2024-09-13T13:50:12.935 INFO:tasks.rados.rados.0.smithi159.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:138: set_chunk oid 39 offset: 1782272 length: 706048 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1782272 snap -1 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:119: finishing set_chunk to oid 20 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:119: done (15 left) 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:139: set_chunk oid 40 offset: 1320448 length: 163840 target oid smithi15938319-40 low_tier offset: 1320448 snap -1 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:124: finishing set_chunk to oid 25 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:124: done (15 left) 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.936 INFO:tasks.rados.rados.0.smithi159.stdout:140: set_chunk oid 41 offset: 1497600 length: 1080320 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1497600 snap -1 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout:134: finishing set_chunk to oid 35 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout:134: done (15 left) 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout:141: set_chunk oid 42 offset: 1781760 length: 467456 target oid smithi15938319-42 low_tier offset: 1781760 snap -1 2024-09-13T13:50:12.937 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:125: finishing set_chunk to oid 26 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:125: done (15 left) 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:142: set_chunk oid 43 offset: 873984 length: 1020928 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 873984 snap -1 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:131: finishing set_chunk to oid 32 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:131: done (15 left) 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout:143: set_chunk oid 44 offset: 1548800 length: 944640 target oid smithi15938319-44 low_tier offset: 1548800 snap -1 2024-09-13T13:50:12.940 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:140: finishing set_chunk to oid 41 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:140: done (15 left) 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:144: set_chunk oid 45 offset: 37888 length: 1147904 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 37888 snap -1 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:136: finishing set_chunk to oid 37 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:136: done (15 left) 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:145: set_chunk oid 46 offset: 587776 length: 949248 target oid smithi15938319-46 low_tier offset: 587776 snap -1 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:138: finishing set_chunk to oid 39 2024-09-13T13:50:12.941 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:138: done (15 left) 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:145 retry set_chunk ! 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:139: finishing set_chunk to oid 40 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:139: done (14 left) 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:128: finishing set_chunk to oid 29 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:143: finishing set_chunk to oid 44 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:128: done (13 left) 2024-09-13T13:50:12.942 INFO:tasks.rados.rados.0.smithi159.stdout:143: done (12 left) 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:145: finishing set_chunk to oid 46 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:145: done (11 left) 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:144: finishing set_chunk to oid 45 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 3 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout:144: done (10 left) 2024-09-13T13:50:12.943 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:270 4.4 4:21c00763:::smithi15938319-21:head [assert-version v6,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.067808569999999999, 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.067831640999999998, 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.984 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 270 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915816+0000", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.88e-06 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915822+0000", 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8769999999999998e-06 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915839+0000", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7663999999999999e-05 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915855+0000", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5627000000000001e-05 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915941+0000", 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5984999999999998e-05 2024-09-13T13:50:12.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.916024+0000", 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3361000000000004e-05 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:282 4.5 4:a1652551:::smithi15938319-33:head [assert-version v6,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.927283+0000", 2024-09-13T13:50:12.988 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.056339499000000001, 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.056399668, 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 282 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927283+0000", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927283+0000", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927284+0000", 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:50:12.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927289+0000", 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7899999999999999e-06 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927301+0000", 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2305e-05 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927310+0000", 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5140000000000001e-06 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927356+0000", 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6025000000000001e-05 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.927406+0000", 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0037999999999998e-05 2024-09-13T13:50:12.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:285 4.5 4:a9f43e9a:::smithi15938319-36:head [assert-version v9,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.929897+0000", 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.053725648000000001, 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053815799999999997, 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.993 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 285 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929897+0000", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929897+0000", 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929898+0000", 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1200000000000001e-06 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929903+0000", 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4689999999999996e-06 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929914+0000", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0297e-05 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929922+0000", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9489999999999996e-06 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929982+0000", 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0124999999999998e-05 2024-09-13T13:50:12.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.930058+0000", 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6377000000000004e-05 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:291 4.4 4:22f57a3c:::smithi15938319-42:head [assert-version v9,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.933095+0000", 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.050527612, 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050647668999999999, 2024-09-13T13:50:12.997 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 291 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933095+0000", 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933095+0000", 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933096+0000", 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.077e-06 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:12.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933100+0000", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3590000000000001e-06 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933110+0000", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4390000000000006e-06 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933115+0000", 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5099999999999998e-06 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933167+0000", 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1659000000000001e-05 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.933248+0000", 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1086000000000003e-05 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.001 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:292 4.5 4:aae6a8d6:::smithi15938319-43:head [assert-version v12,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.934302+0000", 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.049320268, 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.049469017999999997, 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 292 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934302+0000", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934302+0000", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934303+0000", 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8599999999999997e-07 2024-09-13T13:50:13.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934308+0000", 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3260000000000003e-06 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934320+0000", 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1814000000000001e-05 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934328+0000", 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7349999999999996e-06 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934384+0000", 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5958000000000002e-05 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.934430+0000", 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6411e-05 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.006 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-09-13T13:50:13.006 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 5 2024-09-13T13:50:13.006 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:13.051 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:13.050+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout:104: finishing set_chunk to oid 5 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout:104: done (9 left) 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout:133: finishing set_chunk to oid 34 2024-09-13T13:50:13.112 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:13.113 INFO:tasks.rados.rados.0.smithi159.stdout:133: done (8 left) 2024-09-13T13:50:13.113 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:13.113 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.113 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:13.113 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:13.113 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:13.113 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.114 INFO:tasks.rados.rados.0.smithi159.stdout:137: finishing set_chunk to oid 38 2024-09-13T13:50:13.114 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:13.115 INFO:tasks.rados.rados.0.smithi159.stdout:137: done (7 left) 2024-09-13T13:50:13.115 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:13.115 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:50:13.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:13.122 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:13.120+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:13.126 INFO:tasks.rados.rados.0.smithi159.stdout:120: finishing set_chunk to oid 21 2024-09-13T13:50:13.126 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:13.126 INFO:tasks.rados.rados.0.smithi159.stdout:120: done (6 left) 2024-09-13T13:50:13.126 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:132: finishing set_chunk to oid 33 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:141: finishing set_chunk to oid 42 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:135: finishing set_chunk to oid 36 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:132: done (5 left) 2024-09-13T13:50:13.127 INFO:tasks.rados.rados.0.smithi159.stdout:135: done (4 left) 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout:141: done (3 left) 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout:142: finishing set_chunk to oid 43 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout:142: done (2 left) 2024-09-13T13:50:13.128 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:13.222 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:13.222+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.266 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.041743533999998, 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31765422599999998, 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222555+0000", 2024-09-13T13:50:13.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3489999999999999e-06 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222562+0000", 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222572+0000", 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.031e-05 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222600+0000", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7647000000000001e-05 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222658+0000", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8520000000000002e-05 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222671+0000", 2024-09-13T13:50:13.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2588e-05 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538469+0000", 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31579794900000002 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538626+0000", 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000157496 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538781+0000", 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015506799999999999 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.539522+0000", 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074055599999999996 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540140+0000", 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061783300000000003 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540177+0000", 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7011999999999997e-05 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540208+0000", 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0735999999999998e-05 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.273 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.72262701, 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.274 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:50:13.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:50:13.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.766301403, 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015359846, 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.280 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 75 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497995+0000", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.497996+0000", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498055+0000", 2024-09-13T13:50:13.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9426999999999999e-05 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498065+0000", 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8330000000000006e-06 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.498072+0000", 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4080000000000003e-06 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506034+0000", 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079622449999999997 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506082+0000", 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999998e-05 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.506234+0000", 2024-09-13T13:50:13.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015228400000000001 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.512684+0000", 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064494230000000001 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513270+0000", 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058655800000000004 2024-09-13T13:50:13.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513321+0000", 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0655000000000002e-05 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.513355+0000", 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4452999999999997e-05 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.286 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.072210648999999, 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016788053000000001, 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.287 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 93 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192086+0000", 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192087+0000", 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7299999999999996e-07 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192165+0000", 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8444999999999999e-05 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192171+0000", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8919999999999997e-06 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192175+0000", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.929e-06 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199613+0000", 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074384250000000002 2024-09-13T13:50:13.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199676+0000", 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2170999999999998e-05 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.199782+0000", 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010671900000000001 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203302+0000", 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035201080000000001 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208790+0000", 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054877570000000002 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208822+0000", 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1498999999999998e-05 2024-09-13T13:50:13.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208874+0000", 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2534999999999998e-05 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:13.293 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.072115021999998, 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017092758, 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 94 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:13.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192182+0000", 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6400000000000003e-07 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192185+0000", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5359999999999999e-06 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192188+0000", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2179999999999999e-06 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192191+0000", 2024-09-13T13:50:13.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.762e-06 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202584+0000", 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010393502000000001 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202639+0000", 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4867999999999997e-05 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.202723+0000", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4164000000000003e-05 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206571+0000", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003847732 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209193+0000", 2024-09-13T13:50:13.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026220219999999999 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209224+0000", 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1133999999999997e-05 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209275+0000", 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0355999999999997e-05 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.299 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.072099508000001, 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017183035999999999, 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.300 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 95 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192197+0000", 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192198+0000", 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2000000000000001e-07 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192201+0000", 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192204+0000", 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.095e-06 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192207+0000", 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.65e-06 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203324+0000", 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011117171 2024-09-13T13:50:13.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203380+0000", 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6270000000000002e-05 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203467+0000", 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7224000000000004e-05 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206788+0000", 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033212530000000001 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209318+0000", 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002529693 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209353+0000", 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4761000000000001e-05 2024-09-13T13:50:13.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209380+0000", 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.741e-05 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.072086583000001, 2024-09-13T13:50:13.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017280957999999999, 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 96 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:13.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192210+0000", 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192254+0000", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3089e-05 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192258+0000", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6630000000000001e-06 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192262+0000", 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.366e-06 2024-09-13T13:50:13.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203766+0000", 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011504491 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203819+0000", 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2469999999999997e-05 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.203908+0000", 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9716999999999995e-05 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206791+0000", 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002882655 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209425+0000", 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026337750000000001 2024-09-13T13:50:13.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209453+0000", 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8707000000000001e-05 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209491+0000", 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7793999999999998e-05 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.072029841999999, 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017314593999999999, 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 97 2024-09-13T13:50:13.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192267+0000", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2800000000000002e-07 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192270+0000", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3029999999999998e-06 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192274+0000", 2024-09-13T13:50:13.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0539999999999996e-06 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192294+0000", 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0446999999999999e-05 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204411+0000", 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012116511 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204463+0000", 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2281000000000003e-05 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204548+0000", 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4648000000000007e-05 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206793+0000", 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022457190000000002 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209530+0000", 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002736165 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209561+0000", 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1590000000000001e-05 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209582+0000", 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0448000000000001e-05 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.071998556, 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017390893000000001, 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.320 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 98 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192298+0000", 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.29e-07 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192304+0000", 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6339999999999998e-06 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192307+0000", 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0000000000000001e-06 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192310+0000", 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8990000000000001e-06 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204852+0000", 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012542136000000001 2024-09-13T13:50:13.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204907+0000", 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4892999999999998e-05 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.204992+0000", 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5216000000000005e-05 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206796+0000", 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018036059999999999 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209623+0000", 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028274730000000001 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209653+0000", 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0009e-05 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209689+0000", 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5797999999999998e-05 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.071975238, 2024-09-13T13:50:13.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017460854000000001, 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 99 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:13.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192322+0000", 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9200000000000001e-07 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192405+0000", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2977000000000002e-05 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192412+0000", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2629999999999997e-06 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192417+0000", 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1200000000000004e-06 2024-09-13T13:50:13.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205422+0000", 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013005624 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205472+0000", 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0139999999999998e-05 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.205588+0000", 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011588500000000001 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208394+0000", 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028053290000000001 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209732+0000", 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013389560000000001 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209761+0000", 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8484e-05 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209782+0000", 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1484e-05 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.332 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.071872636999998, 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017467890999999999, 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 100 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192424+0000", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192425+0000", 2024-09-13T13:50:13.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7e-07 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192427+0000", 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3910000000000001e-06 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192430+0000", 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2370000000000001e-06 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.192433+0000", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9720000000000001e-06 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206113+0000", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013680159000000001 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206164+0000", 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494999999999997e-05 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.206250+0000", 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6243999999999996e-05 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.208666+0000", 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00241633 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209815+0000", 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001148502 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209852+0000", 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7502000000000001e-05 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.209892+0000", 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9589000000000003e-05 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.338 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.668301120999999, 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.339 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:13.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:50:13.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:50:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:50:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.654922730999999, 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:50:13.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:55 4.2 4:4dc7b2b1:::smithi15938319-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.269938+0000", 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.99435905899999999, 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16908241299999999, 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.350 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 55 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269938+0000", 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269938+0000", 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269938+0000", 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2300000000000002e-07 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269941+0000", 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.672e-06 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269944+0000", 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4419999999999998e-06 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269948+0000", 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2949999999999998e-06 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.270685+0000", 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00073698600000000002 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.270742+0000", 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7231999999999997e-05 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.270823+0000", 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1095999999999997e-05 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.272008+0000", 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011850630000000001 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.438945+0000", 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16693707999999999 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.438980+0000", 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4520000000000002e-05 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.439020+0000", 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0604e-05 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:56 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.269968+0000", 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.99432904300000002, 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.169177728, 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 56 2024-09-13T13:50:13.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269968+0000", 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269968+0000", 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269968+0000", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269970+0000", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269974+0000", 2024-09-13T13:50:13.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0559999999999998e-06 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.269978+0000", 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4520000000000002e-06 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.270983+0000", 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001005007 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.271030+0000", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6987999999999998e-05 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.439045+0000", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16801540500000001 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.439087+0000", 2024-09-13T13:50:13.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2185999999999999e-05 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.439145+0000", 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8096000000000001e-05 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:111 4.2 4:4d382039:::smithi15938319-28:head [setxattr _header (56) in=7b,truncate 2453789] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.443139+0000", 2024-09-13T13:50:13.362 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.82115817700000004, 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.070030393999999996, 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.363 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 111 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443139+0000", 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443139+0000", 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443139+0000", 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5499999999999999e-07 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443141+0000", 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9860000000000001e-06 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443144+0000", 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4680000000000001e-06 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443149+0000", 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6920000000000001e-06 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443731+0000", 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058165199999999999 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443781+0000", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0170999999999998e-05 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443876+0000", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4916999999999999e-05 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.444863+0000", 2024-09-13T13:50:13.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00098711799999999994 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513095+0000", 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.068231775999999994 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513130+0000", 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5281000000000003e-05 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513169+0000", 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8977999999999998e-05 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:112 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.443160+0000", 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.82113680099999997, 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.070154991999999999, 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.369 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 112 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443160+0000", 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443160+0000", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443160+0000", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9900000000000002e-07 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443162+0000", 2024-09-13T13:50:13.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3650000000000001e-06 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443166+0000", 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.997e-06 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.443169+0000", 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1300000000000001e-06 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.444037+0000", 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00086863499999999998 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.444078+0000", 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0895000000000001e-05 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513202+0000", 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.069123243000000001 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513248+0000", 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6054000000000003e-05 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.513315+0000", 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7373999999999996e-05 2024-09-13T13:50:13.374 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:251 4.2 4:4a581da9:::smithi15938319-2:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898066+0000", 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.36623106900000002, 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021858662000000001, 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.375 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 251 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898066+0000", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898066+0000", 2024-09-13T13:50:13.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9860000000000001e-06 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898075+0000", 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.525e-06 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898090+0000", 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4987999999999999e-05 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898099+0000", 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9400000000000008e-06 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4134000000000005e-05 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898265+0000", 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1497999999999998e-05 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917880+0000", 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019614867000000001 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918349+0000", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046954400000000002 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919823+0000", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014737109999999999 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919882+0000", 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8693000000000002e-05 2024-09-13T13:50:13.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919924+0000", 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2775999999999998e-05 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:257 4.4 4:2fda4b5c:::smithi15938319-8:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898132+0000", 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.36616495399999999, 2024-09-13T13:50:13.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015535124000000001, 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 257 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898132+0000", 2024-09-13T13:50:13.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898132+0000", 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898132+0000", 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6000000000000004e-07 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898139+0000", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6209999999999998e-06 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898147+0000", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2269999999999993e-06 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:50:13.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8251000000000002e-05 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898261+0000", 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5736999999999998e-05 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898337+0000", 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5871999999999995e-05 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911586+0000", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013248952 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.912866+0000", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001279887 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913605+0000", 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00073937600000000003 2024-09-13T13:50:13.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913636+0000", 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0768000000000001e-05 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913667+0000", 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0873000000000001e-05 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.387 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:263 4.2 4:4dc7b2b1:::smithi15938319-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.909969+0000", 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.354328011, 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019979912999999998, 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.388 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909969+0000", 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909969+0000", 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909970+0000", 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909979+0000", 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.259e-06 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909994+0000", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5852e-05 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.910003+0000", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0659999999999993e-06 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.910065+0000", 2024-09-13T13:50:13.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1406000000000001e-05 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.910149+0000", 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4148999999999997e-05 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929105+0000", 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018956397999999999 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929696+0000", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059019900000000002 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929882+0000", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186606 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929916+0000", 2024-09-13T13:50:13.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3349000000000002e-05 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929949+0000", 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3157000000000001e-05 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.394 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.395 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:13.394+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:13.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:13.496 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:13.495+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:13.500 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.500 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:13.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.500 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.60224618900000004, 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60227370499999999, 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 1 2024-09-13T13:50:13.505 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.509 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:13.554 INFO:tasks.rados.rados.0.smithi159.stdout:129: finishing set_chunk to oid 30 2024-09-13T13:50:13.554 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:13.554 INFO:tasks.rados.rados.0.smithi159.stdout:129: done (1 left) 2024-09-13T13:50:13.554 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:13.597 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:13.596+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:13.611 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.612 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:13.612 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:13.612 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:13.612 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:13.669 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:13.669+0000 7fc16a180640 -1 osd.11 41 handle_osd_map injecting map crc failure 2024-09-13T13:50:13.670 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:13.668+0000 7f9b0c553640 -1 osd.12 41 handle_osd_map injecting map crc failure 2024-09-13T13:50:13.697 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:13.696+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:91 4.5 e41/34 4:aebdca62:::smithi15938319-23:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.382020+0000", 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.342954918, 2024-09-13T13:50:13.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053901599999999997, 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382020+0000", 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382020+0000", 2024-09-13T13:50:13.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382020+0000", 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382027+0000", 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8390000000000001e-06 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382052+0000", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4604999999999999e-05 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382058+0000", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0950000000000001e-06 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382105+0000", 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7027000000000002e-05 2024-09-13T13:50:13.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.382124+0000", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9609000000000001e-05 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.387382+0000", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052571930000000003 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.387410+0000", 2024-09-13T13:50:13.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8082e-05 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98785865500000003, 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.732 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:50:13.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:50:13.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.738 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98277812799999997, 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024188794, 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742198+0000", 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742205+0000", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742227+0000", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2688999999999999e-05 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742237+0000", 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0151e-05 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742312+0000", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4913999999999995e-05 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for blocked object", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742364+0000", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1774999999999998e-05 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756421+0000", 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014057106999999999 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756508+0000", 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6533999999999995e-05 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766385+0000", 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098774239999999992 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.743 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98261803299999995, 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:50:13.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:50:13.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:50:13.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98256671900000003, 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:50:13.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:50:13.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98249452199999998, 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.757 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.982455517, 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.763 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:50:13.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:213 3.3 3:d7665af5:::smithi15938319-14 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98242800600000002, 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024017457999999998, 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.770 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 213 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742547+0000", 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6800000000000001e-07 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742550+0000", 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.965e-06 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742570+0000", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0207000000000002e-05 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742576+0000", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8889999999999999e-06 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743625+0000", 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001049407 2024-09-13T13:50:13.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743730+0000", 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105025 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.751509+0000", 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077789039999999997 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766480+0000", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014970343000000001 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766513+0000", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3009000000000002e-05 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766540+0000", 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:50:13.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766564+0000", 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3799e-05 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:13.776 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.98239110500000004, 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:50:13.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:50:13.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:201 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.969055581, 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010542467, 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755920+0000", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.513e-06 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755926+0000", 2024-09-13T13:50:13.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5509999999999999e-06 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755953+0000", 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6687999999999999e-05 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755972+0000", 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9550000000000001e-05 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766321+0000", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010349205 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766329+0000", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3479999999999998e-06 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766461+0000", 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132612 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.957041009, 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:13.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:50:13.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:50:13.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:203 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.95053353100000004, 2024-09-13T13:50:13.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019371698999999999, 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5199999999999998e-07 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774444+0000", 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5600000000000001e-06 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793662+0000", 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019218452 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793669+0000", 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793716+0000", 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6953999999999998e-05 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793722+0000", 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793813+0000", 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0705999999999995e-05 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:221 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.93133449599999996, 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018779990999999999, 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793641+0000", 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.316e-06 2024-09-13T13:50:13.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793646+0000", 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7870000000000001e-06 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812292+0000", 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018646276 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812298+0000", 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.152e-06 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812336+0000", 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8336e-05 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812344+0000", 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3909999999999998e-06 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812420+0000", 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5733000000000001e-05 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:234 3.3 3:cdbb9bf3:::smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.92663630200000002, 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021808134999999999, 2024-09-13T13:50:13.802 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 234 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798339+0000", 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1990000000000001e-06 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798344+0000", 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798354+0000", 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8670000000000006e-06 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798359+0000", 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0309999999999998e-06 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798399+0000", 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0191999999999997e-05 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798514+0000", 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114484 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.803569+0000", 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005055492 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808174+0000", 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046051089999999996 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820010+0000", 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011835215 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820079+0000", 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9377999999999999e-05 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820146+0000", 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7336999999999996e-05 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.92648018499999996, 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:50:13.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:50:13.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:234 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.812270+0000", 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.91270403300000003, 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078937069999999998, 2024-09-13T13:50:13.815 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812270+0000", 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812270+0000", 2024-09-13T13:50:13.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812271+0000", 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812276+0000", 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4469999999999999e-06 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819870+0000", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075940979999999996 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819880+0000", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0382e-05 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819974+0000", 2024-09-13T13:50:13.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3706e-05 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819985+0000", 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1270000000000001e-05 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820164+0000", 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000178977 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.819 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.90515359799999995, 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:50:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:50:13.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:250 3.3 3:edb81fa6:::smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.824 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.82686721600000002, 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019084286999999998, 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 250 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898108+0000", 2024-09-13T13:50:13.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6699999999999999e-07 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898111+0000", 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8710000000000001e-06 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898131+0000", 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9939e-05 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898136+0000", 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8599999999999998e-06 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898476+0000", 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033951499999999998 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898641+0000", 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016495999999999999 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909561+0000", 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010919988 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.916236+0000", 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066746660000000001 2024-09-13T13:50:13.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917085+0000", 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084985200000000001 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917156+0000", 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0246999999999993e-05 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917192+0000", 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5921999999999999e-05 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.14.23:14 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.908005+0000", 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.81696953299999997, 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065686520000000003, 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.831 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908005+0000", 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908005+0000", 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908005+0000", 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0800000000000001e-07 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908009+0000", 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2969999999999999e-06 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908055+0000", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6845999999999998e-05 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908060+0000", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8999999999999997e-06 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908689+0000", 2024-09-13T13:50:13.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062824500000000004 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908771+0000", 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2348000000000005e-05 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908946+0000", 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017478099999999999 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911885+0000", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029393850000000001 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.914470+0000", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025845220000000001 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.914523+0000", 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3625999999999998e-05 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.914574+0000", 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0393999999999999e-05 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.15.23:1 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922650+0000", 2024-09-13T13:50:13.837 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.80232401799999997, 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069888570000000002, 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922650+0000", 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:13.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922650+0000", 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922652+0000", 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0310000000000001e-06 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922659+0000", 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2570000000000004e-06 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922673+0000", 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4226999999999999e-05 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922682+0000", 2024-09-13T13:50:13.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6510000000000007e-06 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922744+0000", 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2663000000000006e-05 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922837+0000", 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2694000000000004e-05 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922989+0000", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015246899999999999 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923453+0000", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046329500000000002 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929555+0000", 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061021790000000001 2024-09-13T13:50:13.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929587+0000", 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2391000000000002e-05 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.929639+0000", 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1999999999999997e-05 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:13.843 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:13.844 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:13.851 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:13.843+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:96: done (0 left) 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:146: set_chunk oid 47 offset: 123904 length: 1034752 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 123904 snap -1 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:147: set_chunk oid 48 offset: 843776 length: 765440 target oid smithi15938319-48 low_tier offset: 843776 snap -1 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:148: set_chunk oid 49 offset: 903680 length: 152576 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 903680 snap -1 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:149: set_chunk oid 50 offset: 264192 length: 1471488 target oid smithi15938319-50 low_tier offset: 264192 snap -1 2024-09-13T13:50:13.863 INFO:tasks.rados.rados.0.smithi159.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:150: set_chunk oid 1 offset: 783872 length: 26112 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 783872 snap -1 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:151: set_chunk oid 2 offset: 527872 length: 1171456 target oid smithi15938319-2 low_tier offset: 527872 snap -1 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:152: set_chunk oid 3 offset: 750592 length: 617984 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 750592 snap -1 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:153: set_chunk oid 4 offset: 711168 length: 1155072 target oid smithi15938319-4 low_tier offset: 711168 snap -1 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:154: set_chunk oid 5 offset: 1160704 length: 47616 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1160704 snap -1 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:13.864 INFO:tasks.rados.rados.0.smithi159.stdout:155: set_chunk oid 6 offset: 925184 length: 50176 target oid smithi15938319-6 low_tier offset: 925184 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:156: set_chunk oid 7 offset: 1144320 length: 890880 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1144320 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:157: set_chunk oid 8 offset: 641536 length: 616960 target oid smithi15938319-8 low_tier offset: 641536 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:158: set_chunk oid 9 offset: 768512 length: 551936 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 768512 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:159: set_chunk oid 10 offset: 393216 length: 1001984 target oid smithi15938319-10 low_tier offset: 393216 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:160: set_chunk oid 11 offset: 705024 length: 315904 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 705024 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout:161: set_chunk oid 12 offset: 711680 length: 141312 target oid smithi15938319-12 low_tier offset: 711680 snap -1 2024-09-13T13:50:13.865 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:157: finishing set_chunk to oid 8 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 641536 length: 616960 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:151: finishing set_chunk to oid 2 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 527872 length: 1171456 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:151: done (15 left) 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:157: done (14 left) 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:162: set_chunk oid 13 offset: 1985536 length: 16896 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1985536 snap -1 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout:163: set_chunk oid 14 offset: 16896 length: 344064 target oid smithi15938319-14 low_tier offset: 16896 snap -1 2024-09-13T13:50:13.866 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:159: finishing set_chunk to oid 10 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 393216 length: 1001984 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:155: finishing set_chunk to oid 6 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 925184 length: 50176 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:155: done (15 left) 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:159: done (14 left) 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:164: set_chunk oid 15 offset: 743936 length: 802304 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 743936 snap -1 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout:165: set_chunk oid 16 offset: 626176 length: 692224 target oid smithi15938319-16 low_tier offset: 626176 snap -1 2024-09-13T13:50:13.867 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:154: finishing set_chunk to oid 5 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1160704 length: 47616 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:154: done (15 left) 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:166: set_chunk oid 17 offset: 482816 length: 1465856 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 482816 snap -1 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:158: finishing set_chunk to oid 9 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 768512 length: 551936 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:158: done (15 left) 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:13.868 INFO:tasks.rados.rados.0.smithi159.stdout:167: set_chunk oid 18 offset: 50688 length: 2058240 target oid smithi15938319-18 low_tier offset: 50688 snap -1 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:162: finishing set_chunk to oid 13 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1985536 length: 16896 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:163: finishing set_chunk to oid 14 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 16896 length: 344064 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:162: done (15 left) 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:163: done (14 left) 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:168: set_chunk oid 19 offset: 1920512 length: 281088 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1920512 snap -1 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:13.869 INFO:tasks.rados.rados.0.smithi159.stdout:169: set_chunk oid 20 offset: 1097728 length: 1209856 target oid smithi15938319-20 low_tier offset: 1097728 snap -1 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:164: finishing set_chunk to oid 15 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 743936 length: 802304 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:164: done (15 left) 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:170: set_chunk oid 21 offset: 330752 length: 1517568 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 330752 snap -1 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:167: finishing set_chunk to oid 18 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 50688 length: 2058240 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:167: done (15 left) 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:13.870 INFO:tasks.rados.rados.0.smithi159.stdout:171: set_chunk oid 22 offset: 281600 length: 1256448 target oid smithi15938319-22 low_tier offset: 281600 snap -1 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:166: finishing set_chunk to oid 17 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 482816 length: 1465856 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:169: finishing set_chunk to oid 20 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1097728 length: 1209856 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:166: done (15 left) 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:169: done (14 left) 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:172: set_chunk oid 23 offset: 200704 length: 1504768 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 200704 snap -1 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:13.871 INFO:tasks.rados.rados.0.smithi159.stdout:173: set_chunk oid 24 offset: 180224 length: 2015232 target oid smithi15938319-24 low_tier offset: 180224 snap -1 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:170: finishing set_chunk to oid 21 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 330752 length: 1517568 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:170: done (15 left) 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:174: set_chunk oid 25 offset: 54272 length: 227328 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 54272 snap -1 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:172: finishing set_chunk to oid 23 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 200704 length: 1504768 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:173: finishing set_chunk to oid 24 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 180224 length: 2015232 2024-09-13T13:50:13.872 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:172: done (15 left) 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:173: done (14 left) 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:175: set_chunk oid 26 offset: 1492992 length: 547840 target oid smithi15938319-26 low_tier offset: 1492992 snap -1 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:176: set_chunk oid 27 offset: 552448 length: 14848 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 552448 snap -1 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:152: finishing set_chunk to oid 3 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 750592 length: 617984 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:152: done (15 left) 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:13.873 INFO:tasks.rados.rados.0.smithi159.stdout:177: set_chunk oid 28 offset: 684032 length: 1167872 target oid smithi15938319-28 low_tier offset: 684032 snap -1 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:153: finishing set_chunk to oid 4 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 711168 length: 1155072 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:153: done (15 left) 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:178: set_chunk oid 29 offset: 1655808 length: 805888 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1655808 snap -1 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:156: finishing set_chunk to oid 7 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1144320 length: 890880 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:156: done (15 left) 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:179: set_chunk oid 30 offset: 617984 length: 596480 target oid smithi15938319-30 low_tier offset: 617984 snap -1 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.874 INFO:tasks.rados.rados.0.smithi159.stdout:179: finishing set_chunk to oid 30 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 617984 length: 596480 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:179: done (15 left) 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:180: set_chunk oid 31 offset: 44544 length: 2457600 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 44544 snap -1 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:180: finishing set_chunk to oid 31 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 44544 length: 2457600 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:180: done (15 left) 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout:181: set_chunk oid 32 offset: 1151488 length: 202240 target oid smithi15938319-32 low_tier offset: 1151488 snap -1 2024-09-13T13:50:13.875 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:168: finishing set_chunk to oid 19 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:168: done (15 left) 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:182: set_chunk oid 33 offset: 818176 length: 831488 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 818176 snap -1 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:181: finishing set_chunk to oid 32 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1151488 length: 202240 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:181: done (15 left) 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:183: set_chunk oid 34 offset: 62464 length: 303616 target oid smithi15938319-34 low_tier offset: 62464 snap -1 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.876 INFO:tasks.rados.rados.0.smithi159.stdout:174: finishing set_chunk to oid 25 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:174: done (15 left) 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:184: set_chunk oid 35 offset: 1290752 length: 111104 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1290752 snap -1 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:182: finishing set_chunk to oid 33 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 818176 length: 831488 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:182: done (15 left) 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:185: set_chunk oid 36 offset: 833536 length: 752640 target oid smithi15938319-36 low_tier offset: 833536 snap -1 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.877 INFO:tasks.rados.rados.0.smithi159.stdout:147: finishing set_chunk to oid 48 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:147: done (15 left) 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:186: set_chunk oid 37 offset: 2020864 length: 333312 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2020864 snap -1 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:176: finishing set_chunk to oid 27 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:176: done (15 left) 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:187: set_chunk oid 38 offset: 1968640 length: 309248 target oid smithi15938319-38 low_tier offset: 1968640 snap -1 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:178: finishing set_chunk to oid 29 2024-09-13T13:50:13.878 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:178: done (15 left) 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:188: set_chunk oid 39 offset: 353280 length: 492032 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 353280 snap -1 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:160: finishing set_chunk to oid 11 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:160: done (15 left) 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:189: set_chunk oid 40 offset: 494592 length: 232960 target oid smithi15938319-40 low_tier offset: 494592 snap -1 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:185: finishing set_chunk to oid 36 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 833536 length: 752640 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:13.879 INFO:tasks.rados.rados.0.smithi159.stdout:185: done (15 left) 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:190: set_chunk oid 41 offset: 1389056 length: 998400 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1389056 snap -1 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:149: finishing set_chunk to oid 50 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:149: done (15 left) 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:191: set_chunk oid 42 offset: 818688 length: 424448 target oid smithi15938319-42 low_tier offset: 818688 snap -1 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:175: finishing set_chunk to oid 26 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:175: done (15 left) 2024-09-13T13:50:13.880 INFO:tasks.rados.rados.0.smithi159.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:192: set_chunk oid 43 offset: 437248 length: 1260544 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 437248 snap -1 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:190: finishing set_chunk to oid 41 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1389056 length: 998400 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:190: done (15 left) 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:193: set_chunk oid 44 offset: 223232 length: 1574912 target oid smithi15938319-44 low_tier offset: 223232 snap -1 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:161: finishing set_chunk to oid 12 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:161: done (15 left) 2024-09-13T13:50:13.881 INFO:tasks.rados.rados.0.smithi159.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:194: set_chunk oid 45 offset: 278016 length: 1259520 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 278016 snap -1 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:165: finishing set_chunk to oid 16 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:165: done (15 left) 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:195: set_chunk oid 46 offset: 1388544 length: 1086976 target oid smithi15938319-46 low_tier offset: 1388544 snap -1 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:177: finishing set_chunk to oid 28 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:177: done (15 left) 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:195 retry set_chunk ! 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:183: finishing set_chunk to oid 34 2024-09-13T13:50:13.882 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:183: done (14 left) 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:150: finishing set_chunk to oid 1 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:150: done (13 left) 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:193: finishing set_chunk to oid 44 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 223232 length: 1574912 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:193: done (12 left) 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:194: finishing set_chunk to oid 45 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 278016 length: 1259520 2024-09-13T13:50:13.883 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:194: done (11 left) 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:195: finishing set_chunk to oid 46 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1388544 length: 1086976 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:195: done (10 left) 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:184: finishing set_chunk to oid 35 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:184: done (9 left) 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:192: finishing set_chunk to oid 43 2024-09-13T13:50:13.884 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 437248 length: 1260544 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:192: done (8 left) 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:171: finishing set_chunk to oid 22 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:171: done (7 left) 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:187: finishing set_chunk to oid 38 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:187: done (6 left) 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:188: finishing set_chunk to oid 39 2024-09-13T13:50:13.885 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:188: done (5 left) 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:186: finishing set_chunk to oid 37 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:186: done (4 left) 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:189: finishing set_chunk to oid 40 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout:189: done (3 left) 2024-09-13T13:50:13.886 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:13.945 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:13.945+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:13.953 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:13.953 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:13.953 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:13.953 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:13.954 INFO:tasks.rados.rados.0.smithi159.stdout:191: finishing set_chunk to oid 42 2024-09-13T13:50:13.954 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:13.954 INFO:tasks.rados.rados.0.smithi159.stdout:191: done (2 left) 2024-09-13T13:50:13.954 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:13.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:14.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:14.063 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:14.063 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:14.063 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:14.063 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:14.064 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:14.067 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:14.066+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:14.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:14.088 INFO:tasks.rados.rados.0.smithi159.stdout:148: finishing set_chunk to oid 49 2024-09-13T13:50:14.088 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 3 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.088 INFO:tasks.rados.rados.0.smithi159.stdout:146: finishing set_chunk to oid 47 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:146: done (1 left) 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:148: done (0 left) 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:196: set_chunk oid 47 offset: 1516544 length: 624128 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1516544 snap -1 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:197: set_chunk oid 48 offset: 655872 length: 214528 target oid smithi15938319-48 low_tier offset: 655872 snap -1 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:198: set_chunk oid 49 offset: 961536 length: 633344 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 961536 snap -1 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:199: set_chunk oid 50 offset: 389120 length: 2176000 target oid smithi15938319-50 low_tier offset: 389120 snap -1 2024-09-13T13:50:14.089 INFO:tasks.rados.rados.0.smithi159.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:200: set_chunk oid 1 offset: 1504768 length: 318464 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1504768 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:201: set_chunk oid 2 offset: 18432 length: 1927168 target oid smithi15938319-2 low_tier offset: 18432 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:202: set_chunk oid 3 offset: 110592 length: 1414656 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 110592 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:203: set_chunk oid 4 offset: 689152 length: 1117184 target oid smithi15938319-4 low_tier offset: 689152 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:204: set_chunk oid 5 offset: 1245696 length: 202240 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1245696 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:205: set_chunk oid 6 offset: 27136 length: 329216 target oid smithi15938319-6 low_tier offset: 27136 snap -1 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.090 INFO:tasks.rados.rados.0.smithi159.stdout:206: set_chunk oid 7 offset: 435712 length: 1484800 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 435712 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:207: set_chunk oid 8 offset: 1543168 length: 413696 target oid smithi15938319-8 low_tier offset: 1543168 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:208: set_chunk oid 9 offset: 1025536 length: 692224 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1025536 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:209: set_chunk oid 10 offset: 1781760 length: 167424 target oid smithi15938319-10 low_tier offset: 1781760 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:210: set_chunk oid 11 offset: 25600 length: 2074624 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 25600 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:211: set_chunk oid 12 offset: 398848 length: 957952 target oid smithi15938319-12 low_tier offset: 398848 snap -1 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:197: finishing set_chunk to oid 48 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 655872 length: 214528 2024-09-13T13:50:14.091 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:198: finishing set_chunk to oid 49 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 961536 length: 633344 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:197: done (15 left) 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:198: done (14 left) 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:212: set_chunk oid 13 offset: 2000384 length: 84480 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2000384 snap -1 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:213: set_chunk oid 14 offset: 1253888 length: 538112 target oid smithi15938319-14 low_tier offset: 1253888 snap -1 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:199: finishing set_chunk to oid 50 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 389120 length: 2176000 2024-09-13T13:50:14.092 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:202: finishing set_chunk to oid 3 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 110592 length: 1414656 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:199: done (15 left) 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:202: done (14 left) 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:214: set_chunk oid 15 offset: 1900544 length: 176128 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1900544 snap -1 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:215: set_chunk oid 16 offset: 1359360 length: 479744 target oid smithi15938319-16 low_tier offset: 1359360 snap -1 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:204: finishing set_chunk to oid 5 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1245696 length: 202240 2024-09-13T13:50:14.093 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:209: finishing set_chunk to oid 10 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1781760 length: 167424 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:203: finishing set_chunk to oid 4 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 689152 length: 1117184 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:210: finishing set_chunk to oid 11 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 25600 length: 2074624 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:203: done (15 left) 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:204: done (14 left) 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:209: done (13 left) 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:210: done (12 left) 2024-09-13T13:50:14.094 INFO:tasks.rados.rados.0.smithi159.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:216: set_chunk oid 17 offset: 1633792 length: 546816 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1633792 snap -1 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:217: set_chunk oid 18 offset: 1204224 length: 625152 target oid smithi15938319-18 low_tier offset: 1204224 snap -1 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:218: set_chunk oid 19 offset: 1456128 length: 684544 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1456128 snap -1 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:219: set_chunk oid 20 offset: 51712 length: 679424 target oid smithi15938319-20 low_tier offset: 51712 snap -1 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:212: finishing set_chunk to oid 13 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2000384 length: 84480 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.095 INFO:tasks.rados.rados.0.smithi159.stdout:206: finishing set_chunk to oid 7 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 435712 length: 1484800 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:211: finishing set_chunk to oid 12 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 398848 length: 957952 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:201: finishing set_chunk to oid 2 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 18432 length: 1927168 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:201: done (15 left) 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:206: done (14 left) 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:211: done (13 left) 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:212: done (12 left) 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.096 INFO:tasks.rados.rados.0.smithi159.stdout:220: set_chunk oid 21 offset: 1634304 length: 619008 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1634304 snap -1 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:221: set_chunk oid 22 offset: 299520 length: 294912 target oid smithi15938319-22 low_tier offset: 299520 snap -1 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:222: set_chunk oid 23 offset: 1412096 length: 478208 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1412096 snap -1 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:223: set_chunk oid 24 offset: 589312 length: 1506304 target oid smithi15938319-24 low_tier offset: 589312 snap -1 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:218: finishing set_chunk to oid 19 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1456128 length: 684544 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:215: finishing set_chunk to oid 16 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1359360 length: 479744 2024-09-13T13:50:14.097 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:215: done (15 left) 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:218: done (14 left) 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:224: set_chunk oid 25 offset: 904192 length: 367616 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 904192 snap -1 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:225: set_chunk oid 26 offset: 946688 length: 1243136 target oid smithi15938319-26 low_tier offset: 946688 snap -1 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:220: finishing set_chunk to oid 21 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1634304 length: 619008 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:220: done (15 left) 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.098 INFO:tasks.rados.rados.0.smithi159.stdout:226: set_chunk oid 27 offset: 990208 length: 205312 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 990208 snap -1 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:223: finishing set_chunk to oid 24 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 589312 length: 1506304 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:221: finishing set_chunk to oid 22 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 299520 length: 294912 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:224: finishing set_chunk to oid 25 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 904192 length: 367616 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:221: done (15 left) 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:223: done (14 left) 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:224: done (13 left) 2024-09-13T13:50:14.099 INFO:tasks.rados.rados.0.smithi159.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:227: set_chunk oid 28 offset: 162304 length: 1753600 target oid smithi15938319-28 low_tier offset: 162304 snap -1 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:228: set_chunk oid 29 offset: 1108480 length: 482304 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1108480 snap -1 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:229: set_chunk oid 30 offset: 33792 length: 2118144 target oid smithi15938319-30 low_tier offset: 33792 snap -1 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:225: finishing set_chunk to oid 26 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 946688 length: 1243136 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:225: done (15 left) 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:230: set_chunk oid 31 offset: 1642496 length: 122880 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1642496 snap -1 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.100 INFO:tasks.rados.rados.0.smithi159.stdout:229: finishing set_chunk to oid 30 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 33792 length: 2118144 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:229: done (15 left) 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:231: set_chunk oid 32 offset: 339968 length: 1044992 target oid smithi15938319-32 low_tier offset: 339968 snap -1 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:227: finishing set_chunk to oid 28 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 162304 length: 1753600 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:227: done (15 left) 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:232: set_chunk oid 33 offset: 845312 length: 132096 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 845312 snap -1 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.101 INFO:tasks.rados.rados.0.smithi159.stdout:230: finishing set_chunk to oid 31 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1642496 length: 122880 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:230: done (15 left) 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:233: set_chunk oid 34 offset: 2219520 length: 2048 target oid smithi15938319-34 low_tier offset: 2219520 snap -1 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:232: finishing set_chunk to oid 33 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 845312 length: 132096 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:232: done (15 left) 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:234: set_chunk oid 35 offset: 708608 length: 943104 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 708608 snap -1 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.102 INFO:tasks.rados.rados.0.smithi159.stdout:234: finishing set_chunk to oid 35 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 708608 length: 943104 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:234: done (15 left) 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:235: set_chunk oid 36 offset: 1536 length: 2235904 target oid smithi15938319-36 low_tier offset: 1536 snap -1 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:231: finishing set_chunk to oid 32 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 339968 length: 1044992 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:231: done (15 left) 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:236: set_chunk oid 37 offset: 411136 length: 1459200 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 411136 snap -1 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.103 INFO:tasks.rados.rados.0.smithi159.stdout:235: finishing set_chunk to oid 36 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1536 length: 2235904 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:235: done (15 left) 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:237: set_chunk oid 38 offset: 57344 length: 757248 target oid smithi15938319-38 low_tier offset: 57344 snap -1 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:236: finishing set_chunk to oid 37 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 411136 length: 1459200 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:236: done (15 left) 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.104 INFO:tasks.rados.rados.0.smithi159.stdout:238: set_chunk oid 39 offset: 1901568 length: 721920 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1901568 snap -1 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:238: finishing set_chunk to oid 39 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1901568 length: 721920 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:238: done (15 left) 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:239: set_chunk oid 40 offset: 1611264 length: 1035264 target oid smithi15938319-40 low_tier offset: 1611264 snap -1 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:205: finishing set_chunk to oid 6 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:205: done (15 left) 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:240: set_chunk oid 41 offset: 474112 length: 947200 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 474112 snap -1 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.105 INFO:tasks.rados.rados.0.smithi159.stdout:207: finishing set_chunk to oid 8 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:207: done (15 left) 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:241: set_chunk oid 42 offset: 1967616 length: 586240 target oid smithi15938319-42 low_tier offset: 1967616 snap -1 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:241: finishing set_chunk to oid 42 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1967616 length: 586240 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:241: done (15 left) 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:242: set_chunk oid 43 offset: 1568256 length: 870912 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1568256 snap -1 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.106 INFO:tasks.rados.rados.0.smithi159.stdout:242: finishing set_chunk to oid 43 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1568256 length: 870912 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:242: done (15 left) 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:243: set_chunk oid 44 offset: 1465856 length: 734208 target oid smithi15938319-44 low_tier offset: 1465856 snap -1 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:243: finishing set_chunk to oid 44 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1465856 length: 734208 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:243: done (15 left) 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout:244: set_chunk oid 45 offset: 1903616 length: 807424 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1903616 snap -1 2024-09-13T13:50:14.107 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:217: finishing set_chunk to oid 18 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:217: done (15 left) 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:245: set_chunk oid 46 offset: 570368 length: 2054144 target oid smithi15938319-46 low_tier offset: 570368 snap -1 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:245: finishing set_chunk to oid 46 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 570368 length: 2054144 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:245: done (15 left) 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:245 retry set_chunk ! 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:14.108 INFO:tasks.rados.rados.0.smithi159.stdout:196: finishing set_chunk to oid 47 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:196: done (14 left) 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:216: finishing set_chunk to oid 17 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:216: done (13 left) 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:208: finishing set_chunk to oid 9 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:208: done (12 left) 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:228: finishing set_chunk to oid 29 2024-09-13T13:50:14.109 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:228: done (11 left) 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:200: finishing set_chunk to oid 1 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:200: done (10 left) 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:219: finishing set_chunk to oid 20 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:219: done (9 left) 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:222: finishing set_chunk to oid 23 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.110 INFO:tasks.rados.rados.0.smithi159.stdout:222: done (8 left) 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:214: finishing set_chunk to oid 15 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:214: done (7 left) 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:226: finishing set_chunk to oid 27 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:226: done (6 left) 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:244: finishing set_chunk to oid 45 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 3 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout:244: done (5 left) 2024-09-13T13:50:14.111 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:237: finishing set_chunk to oid 38 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:237: done (4 left) 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:239: finishing set_chunk to oid 40 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:239: done (3 left) 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:240: finishing set_chunk to oid 41 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:240: done (2 left) 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.112 INFO:tasks.rados.rados.0.smithi159.stdout:213: finishing set_chunk to oid 14 2024-09-13T13:50:14.113 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.113 INFO:tasks.rados.rados.0.smithi159.stdout:213: done (1 left) 2024-09-13T13:50:14.113 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.147 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:14.145+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:14.173 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:14.178 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.948690633999998, 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:50:14.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:50:14.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.185 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:9 4.b 4:def5de2b:::smithi15938319-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.140862+0000", 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 2.035457326, 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0094914530000000004, 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 9 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.140862+0000", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.140862+0000", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.140864+0000", 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6029999999999999e-06 2024-09-13T13:50:14.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141506+0000", 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064170999999999998 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141519+0000", 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3164e-05 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141527+0000", 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9990000000000008e-06 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141607+0000", 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0675999999999998e-05 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141612+0000", 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5889999999999996e-06 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.144245+0000", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002633383 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.144336+0000", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0389000000000005e-05 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.144435+0000", 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9480000000000003e-05 2024-09-13T13:50:14.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.148292+0000", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003856553 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150281+0000", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00198957 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150320+0000", 2024-09-13T13:50:14.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8688e-05 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150354+0000", 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3649000000000003e-05 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:10 4.b 4:def5de2b:::smithi15938319-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.192 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.141674+0000", 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 2.0346460149999999, 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088508969999999999, 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 10 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141674+0000", 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141674+0000", 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.141674+0000", 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7899999999999998e-07 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142094+0000", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041934399999999999 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142105+0000", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1516e-05 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142114+0000", 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7409999999999998e-06 2024-09-13T13:50:14.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142173+0000", 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9197999999999998e-05 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142179+0000", 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7989999999999999e-06 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146481+0000", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0043016879999999997 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146574+0000", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3417000000000004e-05 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.146702+0000", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128026 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.149779+0000", 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030770789999999999 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150428+0000", 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064844100000000003 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150483+0000", 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5328999999999998e-05 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150525+0000", 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1640000000000001e-05 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:12 4.b 4:def5de2b:::smithi15938319-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.142169+0000", 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 2.0341503470000002, 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087466650000000007, 2024-09-13T13:50:14.199 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 12 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142169+0000", 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142169+0000", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142170+0000", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142178+0000", 2024-09-13T13:50:14.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.371e-06 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142184+0000", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5709999999999997e-06 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142189+0000", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.994e-06 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142229+0000", 2024-09-13T13:50:14.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9909e-05 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.142232+0000", 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8199999999999998e-06 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.148201+0000", 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00596862 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.148276+0000", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5467000000000001e-05 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150713+0000", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024361220000000002 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150793+0000", 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9964999999999998e-05 2024-09-13T13:50:14.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.150916+0000", 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012346699999999999 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.443952452, 2024-09-13T13:50:14.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:50:14.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.211 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.44390314, 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.212 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:50:14.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.443887436, 2024-09-13T13:50:14.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:14.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:14.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:50:14.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.224 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.44385976, 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.225 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:50:14.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:50:14.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:216 3.2 3:683f8235:::smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.411789017, 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025464730000000001, 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.231 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 216 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764533+0000", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0590000000000001e-06 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764542+0000", 2024-09-13T13:50:14.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4010000000000003e-06 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764562+0000", 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9720000000000001e-05 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764571+0000", 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0110000000000006e-06 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764631+0000", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0647999999999999e-05 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764752+0000", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012058 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774799+0000", 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010046602 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.777512+0000", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027135890000000002 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789910+0000", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012397905000000001 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789952+0000", 2024-09-13T13:50:14.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1604999999999999e-05 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789995+0000", 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3609999999999998e-05 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:14.237 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.408642162, 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:50:14.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:50:14.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.40157318, 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:50:14.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:50:14.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:50:14.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:50:14.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:50:14.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.367034718, 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.252 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:50:14.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:50:14.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.3594048759999999, 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.258 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:50:14.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:50:14.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.264 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.356028504, 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:50:14.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:50:14.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:247 3.2 3:501d6036:::smithi15938319-48 low_tier:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.278285906, 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013184006999999999, 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.271 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 247 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898036+0000", 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5189999999999999e-06 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898049+0000", 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.275e-05 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898073+0000", 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4029999999999999e-05 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898086+0000", 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.256e-05 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898164+0000", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8712000000000001e-05 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898311+0000", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014640599999999999 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.901880+0000", 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035688629999999998 2024-09-13T13:50:14.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.907582+0000", 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057028859999999999 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911105+0000", 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035227000000000001 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0890999999999997e-05 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911218+0000", 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1690000000000001e-05 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:249 3.2 3:5e5aa6e9:::smithi15938319-50 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.277 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.277899345, 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026560466000000001, 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 249 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898421+0000", 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898426+0000", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7740000000000004e-06 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898437+0000", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0868e-05 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898443+0000", 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5409999999999997e-06 2024-09-13T13:50:14.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898782+0000", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033879599999999999 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898917+0000", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013452200000000001 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915774+0000", 2024-09-13T13:50:14.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016857014999999999 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919261+0000", 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034877839999999998 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924925+0000", 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0056640450000000004 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924955+0000", 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9561999999999999e-05 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924981+0000", 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5860999999999999e-05 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.283 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:252 4.b 4:def5de2b:::smithi15938319-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.2778509739999999, 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017224822000000001, 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 252 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.284 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:14.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.06e-07 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898475+0000", 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0560000000000001e-06 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898505+0000", 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9603000000000001e-05 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898511+0000", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1850000000000001e-06 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898567+0000", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5523999999999999e-05 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898640+0000", 2024-09-13T13:50:14.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3752999999999997e-05 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.912256+0000", 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01361593 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.914820+0000", 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025639759999999999 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915602+0000", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078219100000000001 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915651+0000", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8911e-05 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915694+0000", 2024-09-13T13:50:14.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2187000000000001e-05 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:253 4.b 4:d4fa90c1:::smithi15938319-4:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.277778383, 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022974749999999999, 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.290 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 253 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898542+0000", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6000000000000003e-07 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898552+0000", 2024-09-13T13:50:14.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0005e-05 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898562+0000", 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6819999999999998e-06 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898568+0000", 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4400000000000002e-06 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899164+0000", 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059608899999999995 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899245+0000", 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0457999999999994e-05 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918743+0000", 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019497970999999999 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921290+0000", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025477830000000001 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921459+0000", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016888999999999999 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921490+0000", 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0629000000000001e-05 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921516+0000", 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6143e-05 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:256 4.b 4:d7a30b61:::smithi15938319-7:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.277737519, 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016599936999999999, 2024-09-13T13:50:14.296 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 256 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898583+0000", 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1600000000000001e-07 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898586+0000", 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1480000000000002e-06 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898592+0000", 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2790000000000004e-06 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898598+0000", 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899379+0000", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078147899999999996 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899420+0000", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1026999999999999e-05 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911795+0000", 2024-09-13T13:50:14.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012374444 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.912502+0000", 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070700199999999998 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915080+0000", 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002577986 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915136+0000", 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6214000000000001e-05 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915182+0000", 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5957000000000001e-05 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.2531692240000001, 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:14.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:50:14.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:14.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:50:14.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:14.308 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:14.310 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:14.311 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:14.311 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:14.312 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:14.309+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:14.312 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:14.312 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:14.313 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:14.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:14.313 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:14.313 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:14.314 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:14.314 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:14.323 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:14.324 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:14.344 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:14.344 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:14.365 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:14.365 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:14.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:14.386 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:14.408 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:14.408 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:14.410 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:14.410+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:14.428 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:14.428 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:14.430 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:14.430 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:14.439 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:14.453 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:14.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:14.469+0000 7fe5c1132640 0 -- 172.21.15.27:0/19219880 >> v2:172.21.15.122:6808/1255553857 conn(0x7fe5bc21fb00 msgr2=0x7fe5bc221ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:14.484 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:14.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:14.504 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:14.555 INFO:tasks.rados.rados.0.smithi159.stdout:233: finishing set_chunk to oid 34 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:233: done (0 left) 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:246: set_chunk oid 47 offset: 1808384 length: 837120 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1808384 snap -1 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:247: set_chunk oid 48 offset: 989696 length: 1664000 target oid smithi15938319-48 low_tier offset: 989696 snap -1 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:248: set_chunk oid 49 offset: 658432 length: 468992 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 658432 snap -1 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:249: set_chunk oid 50 offset: 930816 length: 261120 target oid smithi15938319-50 low_tier offset: 930816 snap -1 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:250: set_chunk oid 1 offset: 1200640 length: 392192 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1200640 snap -1 2024-09-13T13:50:14.556 INFO:tasks.rados.rados.0.smithi159.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:251: set_chunk oid 2 offset: 1072640 length: 499712 target oid smithi15938319-2 low_tier offset: 1072640 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:252: set_chunk oid 3 offset: 271360 length: 66048 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 271360 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:253: set_chunk oid 4 offset: 200192 length: 702464 target oid smithi15938319-4 low_tier offset: 200192 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:254: set_chunk oid 5 offset: 607744 length: 908288 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 607744 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:255: set_chunk oid 6 offset: 87040 length: 45056 target oid smithi15938319-6 low_tier offset: 87040 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:256: set_chunk oid 7 offset: 937984 length: 409600 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 937984 snap -1 2024-09-13T13:50:14.557 INFO:tasks.rados.rados.0.smithi159.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:257: set_chunk oid 8 offset: 349184 length: 508928 target oid smithi15938319-8 low_tier offset: 349184 snap -1 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:258: set_chunk oid 9 offset: 1280000 length: 585728 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1280000 snap -1 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:259: set_chunk oid 10 offset: 848896 length: 121856 target oid smithi15938319-10 low_tier offset: 848896 snap -1 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:260: set_chunk oid 11 offset: 158208 length: 780800 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 158208 snap -1 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:261: set_chunk oid 12 offset: 702464 length: 15360 target oid smithi15938319-12 low_tier offset: 702464 snap -1 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:247: finishing set_chunk to oid 48 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 989696 length: 1664000 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.558 INFO:tasks.rados.rados.0.smithi159.stdout:247: done (15 left) 2024-09-13T13:50:14.559 INFO:tasks.rados.rados.0.smithi159.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.559 INFO:tasks.rados.rados.0.smithi159.stdout:262: set_chunk oid 13 offset: 260096 length: 198144 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 260096 snap -1 2024-09-13T13:50:14.560 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.560 INFO:tasks.rados.rados.0.smithi159.stdout:246: finishing set_chunk to oid 47 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1808384 length: 837120 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:251: finishing set_chunk to oid 2 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1072640 length: 499712 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:246: done (15 left) 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:251: done (14 left) 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:263: set_chunk oid 14 offset: 295936 length: 603136 target oid smithi15938319-14 low_tier offset: 295936 snap -1 2024-09-13T13:50:14.561 INFO:tasks.rados.rados.0.smithi159.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:264: set_chunk oid 15 offset: 1385984 length: 100352 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1385984 snap -1 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:255: finishing set_chunk to oid 6 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 87040 length: 45056 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:249: finishing set_chunk to oid 50 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 930816 length: 261120 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:257: finishing set_chunk to oid 8 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 349184 length: 508928 2024-09-13T13:50:14.562 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:249: done (15 left) 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:255: done (14 left) 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:257: done (13 left) 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:265: set_chunk oid 16 offset: 356864 length: 1891840 target oid smithi15938319-16 low_tier offset: 356864 snap -1 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:248: finishing set_chunk to oid 49 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 658432 length: 468992 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:248: done (13 left) 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:266: set_chunk oid 17 offset: 1253888 length: 642560 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1253888 snap -1 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.563 INFO:tasks.rados.rados.0.smithi159.stdout:267: set_chunk oid 18 offset: 692736 length: 1188352 target oid smithi15938319-18 low_tier offset: 692736 snap -1 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:268: set_chunk oid 19 offset: 1859072 length: 436736 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1859072 snap -1 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:254: finishing set_chunk to oid 5 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 607744 length: 908288 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:254: done (15 left) 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:269: set_chunk oid 20 offset: 545280 length: 904192 target oid smithi15938319-20 low_tier offset: 545280 snap -1 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:250: finishing set_chunk to oid 1 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1200640 length: 392192 2024-09-13T13:50:14.564 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:261: finishing set_chunk to oid 12 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 702464 length: 15360 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:260: finishing set_chunk to oid 11 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 158208 length: 780800 2024-09-13T13:50:14.565 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:259: finishing set_chunk to oid 10 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 848896 length: 121856 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:258: finishing set_chunk to oid 9 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1280000 length: 585728 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:250: done (15 left) 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:258: done (14 left) 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:259: done (13 left) 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:260: done (12 left) 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:261: done (11 left) 2024-09-13T13:50:14.566 INFO:tasks.rados.rados.0.smithi159.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:270: set_chunk oid 21 offset: 464896 length: 312832 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464896 snap -1 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:271: set_chunk oid 22 offset: 1861632 length: 211456 target oid smithi15938319-22 low_tier offset: 1861632 snap -1 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:272: set_chunk oid 23 offset: 1440768 length: 557056 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1440768 snap -1 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:273: set_chunk oid 24 offset: 8704 length: 1764352 target oid smithi15938319-24 low_tier offset: 8704 snap -1 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:274: set_chunk oid 25 offset: 536576 length: 301056 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 536576 snap -1 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:263: finishing set_chunk to oid 14 2024-09-13T13:50:14.567 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 295936 length: 603136 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:264: finishing set_chunk to oid 15 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1385984 length: 100352 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:265: finishing set_chunk to oid 16 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 356864 length: 1891840 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.568 INFO:tasks.rados.rados.0.smithi159.stdout:263: done (15 left) 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:264: done (14 left) 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:265: done (13 left) 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:275: set_chunk oid 26 offset: 160768 length: 1421824 target oid smithi15938319-26 low_tier offset: 160768 snap -1 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:276: set_chunk oid 27 offset: 1227776 length: 415744 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1227776 snap -1 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:277: set_chunk oid 28 offset: 210944 length: 1430016 target oid smithi15938319-28 low_tier offset: 210944 snap -1 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:268: finishing set_chunk to oid 19 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1859072 length: 436736 2024-09-13T13:50:14.569 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:267: finishing set_chunk to oid 18 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 692736 length: 1188352 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:266: finishing set_chunk to oid 17 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1253888 length: 642560 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:274: finishing set_chunk to oid 25 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 536576 length: 301056 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:266: done (15 left) 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:267: done (14 left) 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:268: done (13 left) 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:274: done (12 left) 2024-09-13T13:50:14.570 INFO:tasks.rados.rados.0.smithi159.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:278: set_chunk oid 29 offset: 510976 length: 1857024 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 510976 snap -1 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:279: set_chunk oid 30 offset: 2168832 length: 182272 target oid smithi15938319-30 low_tier offset: 2168832 snap -1 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:280: set_chunk oid 31 offset: 810496 length: 1494528 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 810496 snap -1 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:281: set_chunk oid 32 offset: 242176 length: 721920 target oid smithi15938319-32 low_tier offset: 242176 snap -1 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:269: finishing set_chunk to oid 20 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 545280 length: 904192 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:270: finishing set_chunk to oid 21 2024-09-13T13:50:14.571 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 464896 length: 312832 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:272: finishing set_chunk to oid 23 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1440768 length: 557056 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:271: finishing set_chunk to oid 22 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1861632 length: 211456 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:277: finishing set_chunk to oid 28 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 210944 length: 1430016 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:275: finishing set_chunk to oid 26 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 160768 length: 1421824 2024-09-13T13:50:14.572 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:269: done (15 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:270: done (14 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:271: done (13 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:272: done (12 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:275: done (11 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:277: done (10 left) 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:282: set_chunk oid 33 offset: 389120 length: 1784320 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 389120 snap -1 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:283: set_chunk oid 34 offset: 283648 length: 1728000 target oid smithi15938319-34 low_tier offset: 283648 snap -1 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:284: set_chunk oid 35 offset: 1561600 length: 800256 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1561600 snap -1 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.573 INFO:tasks.rados.rados.0.smithi159.stdout:285: set_chunk oid 36 offset: 1384448 length: 683520 target oid smithi15938319-36 low_tier offset: 1384448 snap -1 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:286: set_chunk oid 37 offset: 210944 length: 898560 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 210944 snap -1 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:287: set_chunk oid 38 offset: 210944 length: 2357760 target oid smithi15938319-38 low_tier offset: 210944 snap -1 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:278: finishing set_chunk to oid 29 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 510976 length: 1857024 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:280: finishing set_chunk to oid 31 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 810496 length: 1494528 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.574 INFO:tasks.rados.rados.0.smithi159.stdout:278: done (15 left) 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:280: done (14 left) 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:288: set_chunk oid 39 offset: 69120 length: 2207744 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 69120 snap -1 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:289: set_chunk oid 40 offset: 977920 length: 389120 target oid smithi15938319-40 low_tier offset: 977920 snap -1 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:281: finishing set_chunk to oid 32 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 242176 length: 721920 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:282: finishing set_chunk to oid 33 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 389120 length: 1784320 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:281: done (15 left) 2024-09-13T13:50:14.575 INFO:tasks.rados.rados.0.smithi159.stdout:282: done (14 left) 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:290: set_chunk oid 41 offset: 1571328 length: 564224 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1571328 snap -1 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:291: set_chunk oid 42 offset: 883712 length: 233984 target oid smithi15938319-42 low_tier offset: 883712 snap -1 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:284: finishing set_chunk to oid 35 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1561600 length: 800256 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:283: finishing set_chunk to oid 34 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 283648 length: 1728000 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:286: finishing set_chunk to oid 37 2024-09-13T13:50:14.576 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 210944 length: 898560 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:283: done (15 left) 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:284: done (14 left) 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:286: done (13 left) 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:292: set_chunk oid 43 offset: 2195968 length: 251392 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2195968 snap -1 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:293: set_chunk oid 44 offset: 653312 length: 1579008 target oid smithi15938319-44 low_tier offset: 653312 snap -1 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:294: set_chunk oid 45 offset: 466432 length: 2222080 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 466432 snap -1 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:285: finishing set_chunk to oid 36 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1384448 length: 683520 2024-09-13T13:50:14.577 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:285: done (15 left) 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:295: set_chunk oid 46 offset: 674304 length: 1479168 target oid smithi15938319-46 low_tier offset: 674304 snap -1 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:288: finishing set_chunk to oid 39 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 69120 length: 2207744 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:288: done (15 left) 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:296: set_chunk oid 47 offset: 856576 length: 1146880 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 856576 snap -1 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:287: finishing set_chunk to oid 38 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 210944 length: 2357760 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.578 INFO:tasks.rados.rados.0.smithi159.stdout:287: done (15 left) 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:297: set_chunk oid 48 offset: 320000 length: 1412096 target oid smithi15938319-48 low_tier offset: 320000 snap -1 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:291: finishing set_chunk to oid 42 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 883712 length: 233984 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:289: finishing set_chunk to oid 40 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 977920 length: 389120 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:290: finishing set_chunk to oid 41 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1571328 length: 564224 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:289: done (15 left) 2024-09-13T13:50:14.579 INFO:tasks.rados.rados.0.smithi159.stdout:290: done (14 left) 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:291: done (13 left) 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:298: set_chunk oid 49 offset: 840192 length: 1052672 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 840192 snap -1 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:299: set_chunk oid 50 offset: 121344 length: 1457152 target oid smithi15938319-50 low_tier offset: 121344 snap -1 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:300: set_chunk oid 1 offset: 809984 length: 1188352 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 809984 snap -1 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:294: finishing set_chunk to oid 45 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 466432 length: 2222080 2024-09-13T13:50:14.580 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:294: done (15 left) 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:301: set_chunk oid 2 offset: 500224 length: 1202688 target oid smithi15938319-2 low_tier offset: 500224 snap -1 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:296: finishing set_chunk to oid 47 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 856576 length: 1146880 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:296: done (15 left) 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:301 retry set_chunk ! 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:293: finishing set_chunk to oid 44 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 653312 length: 1579008 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.581 INFO:tasks.rados.rados.0.smithi159.stdout:293: done (14 left) 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:295: finishing set_chunk to oid 46 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 674304 length: 1479168 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:297: finishing set_chunk to oid 48 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 320000 length: 1412096 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:295: done (13 left) 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:297: done (12 left) 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:298: finishing set_chunk to oid 49 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 840192 length: 1052672 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.582 INFO:tasks.rados.rados.0.smithi159.stdout:300: finishing set_chunk to oid 1 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 809984 length: 1188352 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:298: done (11 left) 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:300: done (10 left) 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:301: finishing set_chunk to oid 2 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 500224 length: 1202688 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:301: done (9 left) 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:299: finishing set_chunk to oid 50 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 121344 length: 1457152 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.583 INFO:tasks.rados.rados.0.smithi159.stdout:299: done (8 left) 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:273: finishing set_chunk to oid 24 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:273: done (7 left) 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:262: finishing set_chunk to oid 13 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:262: done (6 left) 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:253: finishing set_chunk to oid 4 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 200192 length: 702464 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:256: finishing set_chunk to oid 7 2024-09-13T13:50:14.584 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 937984 length: 409600 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:253: done (5 left) 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:256: done (4 left) 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:252: finishing set_chunk to oid 3 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:252: done (3 left) 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:292: finishing set_chunk to oid 43 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout:292: done (2 left) 2024-09-13T13:50:14.585 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:276: finishing set_chunk to oid 27 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:276: done (1 left) 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:279: finishing set_chunk to oid 30 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:279: done (0 left) 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:302: set_chunk oid 3 offset: 1264128 length: 498688 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1264128 snap -1 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:303: set_chunk oid 4 offset: 931328 length: 250368 target oid smithi15938319-4 low_tier offset: 931328 snap -1 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:304: set_chunk oid 5 offset: 857088 length: 491008 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 857088 snap -1 2024-09-13T13:50:14.586 INFO:tasks.rados.rados.0.smithi159.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:305: set_chunk oid 6 offset: 186368 length: 1573888 target oid smithi15938319-6 low_tier offset: 186368 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:306: set_chunk oid 7 offset: 1401344 length: 528384 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1401344 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:307: set_chunk oid 8 offset: 349184 length: 1680384 target oid smithi15938319-8 low_tier offset: 349184 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:308: set_chunk oid 9 offset: 33280 length: 1887744 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 33280 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:309: set_chunk oid 10 offset: 13312 length: 1073664 target oid smithi15938319-10 low_tier offset: 13312 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:310: set_chunk oid 11 offset: 621056 length: 368128 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 621056 snap -1 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.587 INFO:tasks.rados.rados.0.smithi159.stdout:311: set_chunk oid 12 offset: 30208 length: 1008640 target oid smithi15938319-12 low_tier offset: 30208 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:312: set_chunk oid 13 offset: 574976 length: 540160 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 574976 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:313: set_chunk oid 14 offset: 112128 length: 1995776 target oid smithi15938319-14 low_tier offset: 112128 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:314: set_chunk oid 15 offset: 1663488 length: 378880 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1663488 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:315: set_chunk oid 16 offset: 569856 length: 354816 target oid smithi15938319-16 low_tier offset: 569856 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:316: set_chunk oid 17 offset: 411648 length: 68096 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 411648 snap -1 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.588 INFO:tasks.rados.rados.0.smithi159.stdout:317: set_chunk oid 18 offset: 327168 length: 1475584 target oid smithi15938319-18 low_tier offset: 327168 snap -1 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:302: finishing set_chunk to oid 3 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1264128 length: 498688 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:307: finishing set_chunk to oid 8 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 349184 length: 1680384 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:313: finishing set_chunk to oid 14 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 112128 length: 1995776 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:311: finishing set_chunk to oid 12 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 30208 length: 1008640 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.589 INFO:tasks.rados.rados.0.smithi159.stdout:309: finishing set_chunk to oid 10 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 13312 length: 1073664 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:304: finishing set_chunk to oid 5 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 857088 length: 491008 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:305: finishing set_chunk to oid 6 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 186368 length: 1573888 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:302: done (15 left) 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:304: done (14 left) 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:305: done (13 left) 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:307: done (12 left) 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:309: done (11 left) 2024-09-13T13:50:14.590 INFO:tasks.rados.rados.0.smithi159.stdout:311: done (10 left) 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:313: done (9 left) 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:318: set_chunk oid 19 offset: 536064 length: 1712128 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 536064 snap -1 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:319: set_chunk oid 20 offset: 279040 length: 1155584 target oid smithi15938319-20 low_tier offset: 279040 snap -1 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:303: finishing set_chunk to oid 4 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 931328 length: 250368 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:308: finishing set_chunk to oid 9 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 33280 length: 1887744 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:303: done (10 left) 2024-09-13T13:50:14.591 INFO:tasks.rados.rados.0.smithi159.stdout:308: done (9 left) 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:314: finishing set_chunk to oid 15 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1663488 length: 378880 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:315: finishing set_chunk to oid 16 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 569856 length: 354816 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:317: finishing set_chunk to oid 18 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 327168 length: 1475584 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:320: set_chunk oid 21 offset: 1689600 length: 331776 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1689600 snap -1 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:314: done (9 left) 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:315: done (8 left) 2024-09-13T13:50:14.592 INFO:tasks.rados.rados.0.smithi159.stdout:317: done (7 left) 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:321: set_chunk oid 22 offset: 1218048 length: 733696 target oid smithi15938319-22 low_tier offset: 1218048 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:322: set_chunk oid 23 offset: 912384 length: 96256 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 912384 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:323: set_chunk oid 24 offset: 1983488 length: 183808 target oid smithi15938319-24 low_tier offset: 1983488 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:324: set_chunk oid 25 offset: 423936 length: 916992 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 423936 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:325: set_chunk oid 26 offset: 18432 length: 150528 target oid smithi15938319-26 low_tier offset: 18432 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:326: set_chunk oid 27 offset: 1059328 length: 353792 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1059328 snap -1 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.593 INFO:tasks.rados.rados.0.smithi159.stdout:327: set_chunk oid 28 offset: 387584 length: 549376 target oid smithi15938319-28 low_tier offset: 387584 snap -1 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:328: set_chunk oid 29 offset: 696320 length: 1576448 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 696320 snap -1 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:329: set_chunk oid 30 offset: 1213952 length: 991744 target oid smithi15938319-30 low_tier offset: 1213952 snap -1 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:316: finishing set_chunk to oid 17 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 411648 length: 68096 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:316: done (15 left) 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:330: set_chunk oid 31 offset: 435712 length: 719360 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 435712 snap -1 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:318: finishing set_chunk to oid 19 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 536064 length: 1712128 2024-09-13T13:50:14.594 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:318: done (15 left) 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:331: set_chunk oid 32 offset: 758784 length: 245760 target oid smithi15938319-32 low_tier offset: 758784 snap -1 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:324: finishing set_chunk to oid 25 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 423936 length: 916992 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:326: finishing set_chunk to oid 27 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1059328 length: 353792 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:324: done (15 left) 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:326: done (14 left) 2024-09-13T13:50:14.595 INFO:tasks.rados.rados.0.smithi159.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:332: set_chunk oid 33 offset: 152064 length: 1574912 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 152064 snap -1 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:333: set_chunk oid 34 offset: 97280 length: 2408960 target oid smithi15938319-34 low_tier offset: 97280 snap -1 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:310: finishing set_chunk to oid 11 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 621056 length: 368128 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:321: finishing set_chunk to oid 22 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1218048 length: 733696 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:328: finishing set_chunk to oid 29 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 696320 length: 1576448 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.596 INFO:tasks.rados.rados.0.smithi159.stdout:320: finishing set_chunk to oid 21 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1689600 length: 331776 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:322: finishing set_chunk to oid 23 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 912384 length: 96256 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:329: finishing set_chunk to oid 30 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1213952 length: 991744 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:310: done (15 left) 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:320: done (14 left) 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:321: done (13 left) 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:322: done (12 left) 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:328: done (11 left) 2024-09-13T13:50:14.597 INFO:tasks.rados.rados.0.smithi159.stdout:329: done (10 left) 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:334: set_chunk oid 35 offset: 1832960 length: 571904 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1832960 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:335: set_chunk oid 36 offset: 908800 length: 382976 target oid smithi15938319-36 low_tier offset: 908800 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:336: set_chunk oid 37 offset: 663040 length: 649728 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 663040 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:337: set_chunk oid 38 offset: 275968 length: 1427456 target oid smithi15938319-38 low_tier offset: 275968 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:338: set_chunk oid 39 offset: 761344 length: 1404928 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 761344 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:339: set_chunk oid 40 offset: 869376 length: 292352 target oid smithi15938319-40 low_tier offset: 869376 snap -1 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.598 INFO:tasks.rados.rados.0.smithi159.stdout:327: finishing set_chunk to oid 28 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 387584 length: 549376 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:330: finishing set_chunk to oid 31 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 435712 length: 719360 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:332: finishing set_chunk to oid 33 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 152064 length: 1574912 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:327: done (15 left) 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:330: done (14 left) 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:332: done (13 left) 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.599 INFO:tasks.rados.rados.0.smithi159.stdout:340: set_chunk oid 41 offset: 1012736 length: 884224 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1012736 snap -1 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:341: set_chunk oid 42 offset: 116736 length: 65536 target oid smithi15938319-42 low_tier offset: 116736 snap -1 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:342: set_chunk oid 43 offset: 289792 length: 998400 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 289792 snap -1 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:319: finishing set_chunk to oid 20 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 279040 length: 1155584 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:325: finishing set_chunk to oid 26 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 18432 length: 150528 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:333: finishing set_chunk to oid 34 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 97280 length: 2408960 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.600 INFO:tasks.rados.rados.0.smithi159.stdout:336: finishing set_chunk to oid 37 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 663040 length: 649728 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:335: finishing set_chunk to oid 36 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 908800 length: 382976 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:319: done (15 left) 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:325: done (14 left) 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:333: done (13 left) 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:335: done (12 left) 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:336: done (11 left) 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:343: set_chunk oid 44 offset: 468480 length: 2188288 target oid smithi15938319-44 low_tier offset: 468480 snap -1 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.601 INFO:tasks.rados.rados.0.smithi159.stdout:344: set_chunk oid 45 offset: 138240 length: 69632 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 138240 snap -1 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:345: set_chunk oid 46 offset: 1516544 length: 883712 target oid smithi15938319-46 low_tier offset: 1516544 snap -1 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:346: set_chunk oid 47 offset: 1218560 length: 262656 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1218560 snap -1 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:347: set_chunk oid 48 offset: 2166784 length: 551936 target oid smithi15938319-48 low_tier offset: 2166784 snap -1 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:337: finishing set_chunk to oid 38 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 275968 length: 1427456 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:338: finishing set_chunk to oid 39 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 761344 length: 1404928 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.602 INFO:tasks.rados.rados.0.smithi159.stdout:337: done (15 left) 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:338: done (14 left) 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:348: set_chunk oid 49 offset: 256000 length: 1909760 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 256000 snap -1 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:349: set_chunk oid 50 offset: 1089024 length: 712704 target oid smithi15938319-50 low_tier offset: 1089024 snap -1 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:331: finishing set_chunk to oid 32 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 758784 length: 245760 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:331: done (15 left) 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:350: set_chunk oid 1 offset: 72704 length: 1329664 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 72704 snap -1 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:340: finishing set_chunk to oid 41 2024-09-13T13:50:14.603 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1012736 length: 884224 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:342: finishing set_chunk to oid 43 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 289792 length: 998400 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:340: done (15 left) 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:342: done (14 left) 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:351: set_chunk oid 2 offset: 337920 length: 395264 target oid smithi15938319-2 low_tier offset: 337920 snap -1 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:352: set_chunk oid 3 offset: 607744 length: 343040 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 607744 snap -1 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:344: finishing set_chunk to oid 45 2024-09-13T13:50:14.604 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 138240 length: 69632 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:343: finishing set_chunk to oid 44 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 468480 length: 2188288 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:345: finishing set_chunk to oid 46 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1516544 length: 883712 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:343: done (15 left) 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:344: done (14 left) 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:345: done (13 left) 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:353: set_chunk oid 4 offset: 779264 length: 538112 target oid smithi15938319-4 low_tier offset: 779264 snap -1 2024-09-13T13:50:14.605 INFO:tasks.rados.rados.0.smithi159.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:354: set_chunk oid 5 offset: 811008 length: 861696 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 811008 snap -1 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:348: finishing set_chunk to oid 49 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 256000 length: 1909760 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:348: done (14 left) 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:355: set_chunk oid 6 offset: 483328 length: 702976 target oid smithi15938319-6 low_tier offset: 483328 snap -1 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:355 retry set_chunk ! 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:349: finishing set_chunk to oid 50 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1089024 length: 712704 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout:349: done (14 left) 2024-09-13T13:50:14.606 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:350: finishing set_chunk to oid 1 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 72704 length: 1329664 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:350: done (13 left) 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:352: finishing set_chunk to oid 3 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 607744 length: 343040 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:352: done (12 left) 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:353: finishing set_chunk to oid 4 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 779264 length: 538112 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.607 INFO:tasks.rados.rados.0.smithi159.stdout:353: done (11 left) 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:354: finishing set_chunk to oid 5 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 811008 length: 861696 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:354: done (10 left) 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:355: finishing set_chunk to oid 6 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 483328 length: 702976 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:355: done (9 left) 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:312: finishing set_chunk to oid 13 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.608 INFO:tasks.rados.rados.0.smithi159.stdout:312: done (8 left) 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:341: finishing set_chunk to oid 42 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:341: done (7 left) 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:306: finishing set_chunk to oid 7 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:306: done (6 left) 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:323: finishing set_chunk to oid 24 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout:323: done (5 left) 2024-09-13T13:50:14.609 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:339: finishing set_chunk to oid 40 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:339: done (4 left) 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:334: finishing set_chunk to oid 35 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:334: done (3 left) 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:347: finishing set_chunk to oid 48 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 18 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:347: done (2 left) 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:346: finishing set_chunk to oid 47 2024-09-13T13:50:14.610 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:346: done (1 left) 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:351: finishing set_chunk to oid 2 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:351: done (0 left) 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:356: set_chunk oid 7 offset: 1613312 length: 316928 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1613312 snap -1 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:357: set_chunk oid 8 offset: 1039360 length: 574464 target oid smithi15938319-8 low_tier offset: 1039360 snap -1 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:358: set_chunk oid 9 offset: 276992 length: 1749504 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 276992 snap -1 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.611 INFO:tasks.rados.rados.0.smithi159.stdout:359: set_chunk oid 10 offset: 630784 length: 371712 target oid smithi15938319-10 low_tier offset: 630784 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:360: set_chunk oid 11 offset: 1029632 length: 59392 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1029632 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:361: set_chunk oid 12 offset: 103424 length: 745472 target oid smithi15938319-12 low_tier offset: 103424 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:362: set_chunk oid 13 offset: 934912 length: 251392 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 934912 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:363: set_chunk oid 14 offset: 787456 length: 627712 target oid smithi15938319-14 low_tier offset: 787456 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:364: set_chunk oid 15 offset: 1024 length: 1253888 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1024 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:365: set_chunk oid 16 offset: 831488 length: 921088 target oid smithi15938319-16 low_tier offset: 831488 snap -1 2024-09-13T13:50:14.612 INFO:tasks.rados.rados.0.smithi159.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:366: set_chunk oid 17 offset: 1393152 length: 830464 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1393152 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:367: set_chunk oid 18 offset: 415744 length: 379392 target oid smithi15938319-18 low_tier offset: 415744 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:368: set_chunk oid 19 offset: 1754624 length: 440320 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1754624 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:369: set_chunk oid 20 offset: 827904 length: 147968 target oid smithi15938319-20 low_tier offset: 827904 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:370: set_chunk oid 21 offset: 698880 length: 441856 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 698880 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout:371: set_chunk oid 22 offset: 1930752 length: 129024 target oid smithi15938319-22 low_tier offset: 1930752 snap -1 2024-09-13T13:50:14.613 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:359: finishing set_chunk to oid 10 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 630784 length: 371712 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:359: done (15 left) 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:372: set_chunk oid 23 offset: 1358848 length: 289792 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1358848 snap -1 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:361: finishing set_chunk to oid 12 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 103424 length: 745472 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:356: finishing set_chunk to oid 7 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1613312 length: 316928 2024-09-13T13:50:14.614 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:357: finishing set_chunk to oid 8 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1039360 length: 574464 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:358: finishing set_chunk to oid 9 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 276992 length: 1749504 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:363: finishing set_chunk to oid 14 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 787456 length: 627712 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:365: finishing set_chunk to oid 16 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 831488 length: 921088 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.615 INFO:tasks.rados.rados.0.smithi159.stdout:356: done (15 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:357: done (14 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:358: done (13 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:361: done (12 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:363: done (11 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:365: done (10 left) 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:373: set_chunk oid 24 offset: 919552 length: 658432 target oid smithi15938319-24 low_tier offset: 919552 snap -1 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:362: finishing set_chunk to oid 13 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 934912 length: 251392 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:374: set_chunk oid 25 offset: 311808 length: 916992 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 311808 snap -1 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:364: finishing set_chunk to oid 15 2024-09-13T13:50:14.616 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1024 length: 1253888 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:370: finishing set_chunk to oid 21 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 698880 length: 441856 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:362: done (11 left) 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:364: done (10 left) 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:370: done (9 left) 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:375: set_chunk oid 26 offset: 1154048 length: 681472 target oid smithi15938319-26 low_tier offset: 1154048 snap -1 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:376: set_chunk oid 27 offset: 1281536 length: 848896 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1281536 snap -1 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.617 INFO:tasks.rados.rados.0.smithi159.stdout:377: set_chunk oid 28 offset: 1373696 length: 314880 target oid smithi15938319-28 low_tier offset: 1373696 snap -1 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:378: set_chunk oid 29 offset: 52736 length: 2014208 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 52736 snap -1 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:366: finishing set_chunk to oid 17 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1393152 length: 830464 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:368: finishing set_chunk to oid 19 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1754624 length: 440320 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:371: finishing set_chunk to oid 22 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1930752 length: 129024 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:372: finishing set_chunk to oid 23 2024-09-13T13:50:14.618 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1358848 length: 289792 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:366: done (12 left) 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:368: done (11 left) 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:371: done (10 left) 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:372: done (9 left) 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:379: set_chunk oid 30 offset: 118272 length: 604672 target oid smithi15938319-30 low_tier offset: 118272 snap -1 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:380: set_chunk oid 31 offset: 1158144 length: 817152 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1158144 snap -1 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:381: set_chunk oid 32 offset: 1960960 length: 487424 target oid smithi15938319-32 low_tier offset: 1960960 snap -1 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.619 INFO:tasks.rados.rados.0.smithi159.stdout:382: set_chunk oid 33 offset: 52736 length: 1122816 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 52736 snap -1 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:383: set_chunk oid 34 offset: 429568 length: 68096 target oid smithi15938319-34 low_tier offset: 429568 snap -1 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:384: set_chunk oid 35 offset: 801792 length: 1091072 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 801792 snap -1 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:385: set_chunk oid 36 offset: 1289728 length: 757248 target oid smithi15938319-36 low_tier offset: 1289728 snap -1 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:373: finishing set_chunk to oid 24 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 919552 length: 658432 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:374: finishing set_chunk to oid 25 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 311808 length: 916992 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.620 INFO:tasks.rados.rados.0.smithi159.stdout:376: finishing set_chunk to oid 27 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1281536 length: 848896 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:375: finishing set_chunk to oid 26 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1154048 length: 681472 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:377: finishing set_chunk to oid 28 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1373696 length: 314880 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:380: finishing set_chunk to oid 31 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1158144 length: 817152 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:379: finishing set_chunk to oid 30 2024-09-13T13:50:14.621 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 118272 length: 604672 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:373: done (15 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:374: done (14 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:375: done (13 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:376: done (12 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:377: done (11 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:379: done (10 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:380: done (9 left) 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:386: set_chunk oid 37 offset: 2250240 length: 100352 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2250240 snap -1 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:387: set_chunk oid 38 offset: 1391616 length: 983552 target oid smithi15938319-38 low_tier offset: 1391616 snap -1 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:388: set_chunk oid 39 offset: 552448 length: 1181184 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 552448 snap -1 2024-09-13T13:50:14.622 INFO:tasks.rados.rados.0.smithi159.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:389: set_chunk oid 40 offset: 391168 length: 1767424 target oid smithi15938319-40 low_tier offset: 391168 snap -1 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:390: set_chunk oid 41 offset: 96768 length: 1080832 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 96768 snap -1 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:391: set_chunk oid 42 offset: 869888 length: 402432 target oid smithi15938319-42 low_tier offset: 869888 snap -1 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:392: set_chunk oid 43 offset: 1928192 length: 768000 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1928192 snap -1 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:382: finishing set_chunk to oid 33 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 52736 length: 1122816 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:382: done (15 left) 2024-09-13T13:50:14.623 INFO:tasks.rados.rados.0.smithi159.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:393: set_chunk oid 44 offset: 552448 length: 926208 target oid smithi15938319-44 low_tier offset: 552448 snap -1 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:385: finishing set_chunk to oid 36 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1289728 length: 757248 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:386: finishing set_chunk to oid 37 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2250240 length: 100352 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:385: done (15 left) 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:386: done (14 left) 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:388: finishing set_chunk to oid 39 2024-09-13T13:50:14.624 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 552448 length: 1181184 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:394: set_chunk oid 45 offset: 1707520 length: 777728 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1707520 snap -1 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:388: done (14 left) 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:395: set_chunk oid 46 offset: 1918464 length: 797184 target oid smithi15938319-46 low_tier offset: 1918464 snap -1 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:396: set_chunk oid 47 offset: 2232832 length: 292352 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2232832 snap -1 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:387: finishing set_chunk to oid 38 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1391616 length: 983552 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:392: finishing set_chunk to oid 43 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1928192 length: 768000 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.625 INFO:tasks.rados.rados.0.smithi159.stdout:391: finishing set_chunk to oid 42 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 869888 length: 402432 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:390: finishing set_chunk to oid 41 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 96768 length: 1080832 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:387: done (15 left) 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:390: done (14 left) 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:391: done (13 left) 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:392: done (12 left) 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:397: set_chunk oid 48 offset: 804352 length: 1446912 target oid smithi15938319-48 low_tier offset: 804352 snap -1 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:389: finishing set_chunk to oid 40 2024-09-13T13:50:14.626 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 391168 length: 1767424 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:389: done (12 left) 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:398: set_chunk oid 49 offset: 588288 length: 2176512 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 588288 snap -1 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:399: set_chunk oid 50 offset: 531456 length: 5632 target oid smithi15938319-50 low_tier offset: 531456 snap -1 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:400: set_chunk oid 1 offset: 570880 length: 728576 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 570880 snap -1 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:401: set_chunk oid 2 offset: 1193984 length: 580096 target oid smithi15938319-2 low_tier offset: 1193984 snap -1 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:398: finishing set_chunk to oid 49 2024-09-13T13:50:14.627 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 588288 length: 2176512 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:398: done (15 left) 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:402: set_chunk oid 3 offset: 623104 length: 1263616 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 623104 snap -1 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:399: finishing set_chunk to oid 50 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 531456 length: 5632 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:400: finishing set_chunk to oid 1 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 570880 length: 728576 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:399: done (15 left) 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:400: done (14 left) 2024-09-13T13:50:14.628 INFO:tasks.rados.rados.0.smithi159.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:403: set_chunk oid 4 offset: 1143296 length: 252416 target oid smithi15938319-4 low_tier offset: 1143296 snap -1 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:404: set_chunk oid 5 offset: 238080 length: 1553920 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 238080 snap -1 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:401: finishing set_chunk to oid 2 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1193984 length: 580096 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:401: done (15 left) 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:405: set_chunk oid 6 offset: 298496 length: 782336 target oid smithi15938319-6 low_tier offset: 298496 snap -1 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:404: finishing set_chunk to oid 5 2024-09-13T13:50:14.629 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 238080 length: 1553920 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:402: finishing set_chunk to oid 3 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 623104 length: 1263616 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:402: done (15 left) 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:404: done (14 left) 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:406: set_chunk oid 7 offset: 642560 length: 641536 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 642560 snap -1 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:407: set_chunk oid 8 offset: 1268224 length: 298496 target oid smithi15938319-8 low_tier offset: 1268224 snap -1 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:405: finishing set_chunk to oid 6 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 298496 length: 782336 2024-09-13T13:50:14.630 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:405: done (15 left) 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:408: set_chunk oid 9 offset: 761856 length: 646144 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 761856 snap -1 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:403: finishing set_chunk to oid 4 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1143296 length: 252416 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:403: done (15 left) 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:409: set_chunk oid 10 offset: 494592 length: 64000 target oid smithi15938319-10 low_tier offset: 494592 snap -1 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:406: finishing set_chunk to oid 7 2024-09-13T13:50:14.631 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 642560 length: 641536 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:406: done (15 left) 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:409 retry set_chunk ! 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:407: finishing set_chunk to oid 8 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1268224 length: 298496 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:407: done (14 left) 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 14 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:408: finishing set_chunk to oid 9 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 761856 length: 646144 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.632 INFO:tasks.rados.rados.0.smithi159.stdout:408: done (13 left) 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:393: finishing set_chunk to oid 44 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:393: done (12 left) 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:367: finishing set_chunk to oid 18 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:367: done (11 left) 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:378: finishing set_chunk to oid 29 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 52736 length: 2014208 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:394: finishing set_chunk to oid 45 2024-09-13T13:50:14.633 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1707520 length: 777728 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:378: done (10 left) 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:394: done (9 left) 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:384: finishing set_chunk to oid 35 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 801792 length: 1091072 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:384: done (8 left) 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:397: finishing set_chunk to oid 48 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 804352 length: 1446912 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout:397: done (7 left) 2024-09-13T13:50:14.634 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:360: finishing set_chunk to oid 11 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:360: done (6 left) 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:369: finishing set_chunk to oid 20 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:369: done (5 left) 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:381: finishing set_chunk to oid 32 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:381: done (4 left) 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.635 INFO:tasks.rados.rados.0.smithi159.stdout:395: finishing set_chunk to oid 46 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:395: done (3 left) 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:396: finishing set_chunk to oid 47 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:396: done (2 left) 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:383: finishing set_chunk to oid 34 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:383: done (1 left) 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:409: finishing set_chunk to oid 10 2024-09-13T13:50:14.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:409: done (0 left) 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:410: set_chunk oid 11 offset: 1075200 length: 810496 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1075200 snap -1 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:411: set_chunk oid 12 offset: 1094656 length: 78336 target oid smithi15938319-12 low_tier offset: 1094656 snap -1 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:412: set_chunk oid 13 offset: 799232 length: 1128448 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 799232 snap -1 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:413: set_chunk oid 14 offset: 369152 length: 392704 target oid smithi15938319-14 low_tier offset: 369152 snap -1 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:414: set_chunk oid 15 offset: 1796608 length: 416256 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1796608 snap -1 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.637 INFO:tasks.rados.rados.0.smithi159.stdout:415: set_chunk oid 16 offset: 1898496 length: 3072 target oid smithi15938319-16 low_tier offset: 1898496 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:416: set_chunk oid 17 offset: 1479680 length: 559104 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1479680 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:417: set_chunk oid 18 offset: 1378304 length: 501248 target oid smithi15938319-18 low_tier offset: 1378304 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:418: set_chunk oid 19 offset: 546304 length: 50688 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 546304 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:419: set_chunk oid 20 offset: 605184 length: 1360384 target oid smithi15938319-20 low_tier offset: 605184 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:420: set_chunk oid 21 offset: 632832 length: 964608 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 632832 snap -1 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.638 INFO:tasks.rados.rados.0.smithi159.stdout:421: set_chunk oid 22 offset: 203264 length: 108544 target oid smithi15938319-22 low_tier offset: 203264 snap -1 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:422: set_chunk oid 23 offset: 1028608 length: 651776 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1028608 snap -1 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:423: set_chunk oid 24 offset: 940544 length: 1421312 target oid smithi15938319-24 low_tier offset: 940544 snap -1 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:424: set_chunk oid 25 offset: 729600 length: 1385984 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 729600 snap -1 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:425: set_chunk oid 26 offset: 64512 length: 1696256 target oid smithi15938319-26 low_tier offset: 64512 snap -1 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:415: finishing set_chunk to oid 16 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1898496 length: 3072 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.639 INFO:tasks.rados.rados.0.smithi159.stdout:415: done (15 left) 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:426: set_chunk oid 27 offset: 11776 length: 1576448 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 11776 snap -1 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:413: finishing set_chunk to oid 14 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 369152 length: 392704 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:411: finishing set_chunk to oid 12 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1094656 length: 78336 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:410: finishing set_chunk to oid 11 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1075200 length: 810496 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:422: finishing set_chunk to oid 23 2024-09-13T13:50:14.640 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1028608 length: 651776 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:420: finishing set_chunk to oid 21 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 632832 length: 964608 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:414: finishing set_chunk to oid 15 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1796608 length: 416256 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:417: finishing set_chunk to oid 18 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1378304 length: 501248 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:418: finishing set_chunk to oid 19 2024-09-13T13:50:14.641 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 546304 length: 50688 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:421: finishing set_chunk to oid 22 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 203264 length: 108544 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:416: finishing set_chunk to oid 17 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1479680 length: 559104 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:410: done (15 left) 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:411: done (14 left) 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:413: done (13 left) 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:414: done (12 left) 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:416: done (11 left) 2024-09-13T13:50:14.642 INFO:tasks.rados.rados.0.smithi159.stdout:417: done (10 left) 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:418: done (9 left) 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:420: done (8 left) 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:421: done (7 left) 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:422: done (6 left) 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:427: set_chunk oid 28 offset: 1180672 length: 185344 target oid smithi15938319-28 low_tier offset: 1180672 snap -1 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:424: finishing set_chunk to oid 25 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 729600 length: 1385984 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:423: finishing set_chunk to oid 24 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 940544 length: 1421312 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:412: finishing set_chunk to oid 13 2024-09-13T13:50:14.643 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 799232 length: 1128448 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:412: done (6 left) 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:423: done (5 left) 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:424: done (4 left) 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:428: set_chunk oid 29 offset: 775168 length: 241664 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 775168 snap -1 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:429: set_chunk oid 30 offset: 1466368 length: 794112 target oid smithi15938319-30 low_tier offset: 1466368 snap -1 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:430: set_chunk oid 31 offset: 532480 length: 1597440 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 532480 snap -1 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:431: set_chunk oid 32 offset: 253440 length: 879104 target oid smithi15938319-32 low_tier offset: 253440 snap -1 2024-09-13T13:50:14.644 INFO:tasks.rados.rados.0.smithi159.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:432: set_chunk oid 33 offset: 702464 length: 204800 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 702464 snap -1 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:433: set_chunk oid 34 offset: 253440 length: 1559040 target oid smithi15938319-34 low_tier offset: 253440 snap -1 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:419: finishing set_chunk to oid 20 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 605184 length: 1360384 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:426: finishing set_chunk to oid 27 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 11776 length: 1576448 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:425: finishing set_chunk to oid 26 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 64512 length: 1696256 2024-09-13T13:50:14.645 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:427: finishing set_chunk to oid 28 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1180672 length: 185344 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:434: set_chunk oid 35 offset: 1016320 length: 1127424 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1016320 snap -1 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:419: done (10 left) 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:425: done (9 left) 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:426: done (8 left) 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:427: done (7 left) 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:435: set_chunk oid 36 offset: 1165312 length: 1308160 target oid smithi15938319-36 low_tier offset: 1165312 snap -1 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:436: set_chunk oid 37 offset: 181248 length: 1382400 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 181248 snap -1 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.646 INFO:tasks.rados.rados.0.smithi159.stdout:437: set_chunk oid 38 offset: 1425408 length: 180736 target oid smithi15938319-38 low_tier offset: 1425408 snap -1 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:438: set_chunk oid 39 offset: 389632 length: 227328 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 389632 snap -1 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:429: finishing set_chunk to oid 30 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1466368 length: 794112 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:430: finishing set_chunk to oid 31 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 532480 length: 1597440 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:429: done (10 left) 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:430: done (9 left) 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.647 INFO:tasks.rados.rados.0.smithi159.stdout:439: set_chunk oid 40 offset: 2427904 length: 56320 target oid smithi15938319-40 low_tier offset: 2427904 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:440: set_chunk oid 41 offset: 597504 length: 352256 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 597504 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:441: set_chunk oid 42 offset: 1184768 length: 297472 target oid smithi15938319-42 low_tier offset: 1184768 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:442: set_chunk oid 43 offset: 340480 length: 1741312 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 340480 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:443: set_chunk oid 44 offset: 602112 length: 1479680 target oid smithi15938319-44 low_tier offset: 602112 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:444: set_chunk oid 45 offset: 121856 length: 443904 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 121856 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout:445: set_chunk oid 46 offset: 2032128 length: 470528 target oid smithi15938319-46 low_tier offset: 2032128 snap -1 2024-09-13T13:50:14.648 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:431: finishing set_chunk to oid 32 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 253440 length: 879104 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:432: finishing set_chunk to oid 33 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 702464 length: 204800 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:433: finishing set_chunk to oid 34 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 253440 length: 1559040 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:434: finishing set_chunk to oid 35 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1016320 length: 1127424 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:438: finishing set_chunk to oid 39 2024-09-13T13:50:14.649 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 389632 length: 227328 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:436: finishing set_chunk to oid 37 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 181248 length: 1382400 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:431: done (15 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:432: done (14 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:433: done (13 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:434: done (12 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:436: done (11 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:438: done (10 left) 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:446: set_chunk oid 47 offset: 203776 length: 711680 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 203776 snap -1 2024-09-13T13:50:14.650 INFO:tasks.rados.rados.0.smithi159.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:447: set_chunk oid 48 offset: 697856 length: 1138688 target oid smithi15938319-48 low_tier offset: 697856 snap -1 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:448: set_chunk oid 49 offset: 1442304 length: 1257984 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1442304 snap -1 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:449: set_chunk oid 50 offset: 1639936 length: 935936 target oid smithi15938319-50 low_tier offset: 1639936 snap -1 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:450: set_chunk oid 1 offset: 169984 length: 165888 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 169984 snap -1 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:451: set_chunk oid 2 offset: 177152 length: 685056 target oid smithi15938319-2 low_tier offset: 177152 snap -1 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:435: finishing set_chunk to oid 36 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1165312 length: 1308160 2024-09-13T13:50:14.651 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:435: done (15 left) 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:452: set_chunk oid 3 offset: 169472 length: 1422848 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 169472 snap -1 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:441: finishing set_chunk to oid 42 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1184768 length: 297472 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:439: finishing set_chunk to oid 40 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 2427904 length: 56320 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:440: finishing set_chunk to oid 41 2024-09-13T13:50:14.652 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 597504 length: 352256 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:442: finishing set_chunk to oid 43 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 340480 length: 1741312 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:439: done (15 left) 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:440: done (14 left) 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:441: done (13 left) 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:442: done (12 left) 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:453: set_chunk oid 4 offset: 212480 length: 1185792 target oid smithi15938319-4 low_tier offset: 212480 snap -1 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:454: set_chunk oid 5 offset: 799744 length: 748544 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 799744 snap -1 2024-09-13T13:50:14.653 INFO:tasks.rados.rados.0.smithi159.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:455: set_chunk oid 6 offset: 684544 length: 337408 target oid smithi15938319-6 low_tier offset: 684544 snap -1 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:456: set_chunk oid 7 offset: 1447424 length: 41984 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1447424 snap -1 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:444: finishing set_chunk to oid 45 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 121856 length: 443904 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:445: finishing set_chunk to oid 46 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 2032128 length: 470528 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:444: done (15 left) 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:445: done (14 left) 2024-09-13T13:50:14.654 INFO:tasks.rados.rados.0.smithi159.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:457: set_chunk oid 8 offset: 1088512 length: 655360 target oid smithi15938319-8 low_tier offset: 1088512 snap -1 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:458: set_chunk oid 9 offset: 482304 length: 1335808 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 482304 snap -1 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:447: finishing set_chunk to oid 48 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 697856 length: 1138688 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:447: done (15 left) 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:459: set_chunk oid 10 offset: 733696 length: 527360 target oid smithi15938319-10 low_tier offset: 733696 snap -1 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:443: finishing set_chunk to oid 44 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 602112 length: 1479680 2024-09-13T13:50:14.655 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:451: finishing set_chunk to oid 2 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 177152 length: 685056 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:449: finishing set_chunk to oid 50 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1639936 length: 935936 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:452: finishing set_chunk to oid 3 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 169472 length: 1422848 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:443: done (15 left) 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:449: done (14 left) 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:451: done (13 left) 2024-09-13T13:50:14.656 INFO:tasks.rados.rados.0.smithi159.stdout:452: done (12 left) 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:460: set_chunk oid 11 offset: 848896 length: 823296 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 848896 snap -1 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:461: set_chunk oid 12 offset: 527360 length: 1257472 target oid smithi15938319-12 low_tier offset: 527360 snap -1 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:462: set_chunk oid 13 offset: 1193472 length: 169472 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1193472 snap -1 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:463: set_chunk oid 14 offset: 166912 length: 1410560 target oid smithi15938319-14 low_tier offset: 166912 snap -1 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:446: finishing set_chunk to oid 47 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 203776 length: 711680 2024-09-13T13:50:14.657 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:446: done (15 left) 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:464: set_chunk oid 15 offset: 1318400 length: 843264 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1318400 snap -1 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:453: finishing set_chunk to oid 4 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 212480 length: 1185792 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:453: done (15 left) 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:465: set_chunk oid 16 offset: 1500160 length: 142848 target oid smithi15938319-16 low_tier offset: 1500160 snap -1 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:457: finishing set_chunk to oid 8 2024-09-13T13:50:14.658 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1088512 length: 655360 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:455: finishing set_chunk to oid 6 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 684544 length: 337408 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:455: done (15 left) 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:457: done (14 left) 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:460: finishing set_chunk to oid 11 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 848896 length: 823296 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:458: finishing set_chunk to oid 9 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 482304 length: 1335808 2024-09-13T13:50:14.659 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:466: set_chunk oid 17 offset: 669696 length: 419840 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 669696 snap -1 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:454: finishing set_chunk to oid 5 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 799744 length: 748544 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:454: done (14 left) 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:458: done (13 left) 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:460: done (12 left) 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:467: set_chunk oid 18 offset: 1420288 length: 148992 target oid smithi15938319-18 low_tier offset: 1420288 snap -1 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:459: finishing set_chunk to oid 10 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 733696 length: 527360 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:463: finishing set_chunk to oid 14 2024-09-13T13:50:14.660 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 166912 length: 1410560 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:461: finishing set_chunk to oid 12 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 527360 length: 1257472 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:459: done (12 left) 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:461: done (11 left) 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:463: done (10 left) 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:468: set_chunk oid 19 offset: 659968 length: 744448 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 659968 snap -1 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:469: set_chunk oid 20 offset: 2018304 length: 135168 target oid smithi15938319-20 low_tier offset: 2018304 snap -1 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.661 INFO:tasks.rados.rados.0.smithi159.stdout:470: set_chunk oid 21 offset: 1299456 length: 187392 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1299456 snap -1 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:471: set_chunk oid 22 offset: 999424 length: 108032 target oid smithi15938319-22 low_tier offset: 999424 snap -1 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:472: set_chunk oid 23 offset: 468480 length: 321536 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 468480 snap -1 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:465: finishing set_chunk to oid 16 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1500160 length: 142848 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:473: set_chunk oid 24 offset: 181760 length: 1433600 target oid smithi15938319-24 low_tier offset: 181760 snap -1 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:465: done (15 left) 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:474: set_chunk oid 25 offset: 1378304 length: 523776 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1378304 snap -1 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.662 INFO:tasks.rados.rados.0.smithi159.stdout:464: finishing set_chunk to oid 15 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1318400 length: 843264 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:456: finishing set_chunk to oid 7 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1447424 length: 41984 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:468: finishing set_chunk to oid 19 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 659968 length: 744448 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:456: done (15 left) 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:464: done (14 left) 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:468: done (13 left) 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.663 INFO:tasks.rados.rados.0.smithi159.stdout:466: finishing set_chunk to oid 17 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 669696 length: 419840 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:475: set_chunk oid 26 offset: 1567744 length: 78848 target oid smithi15938319-26 low_tier offset: 1567744 snap -1 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:466: done (13 left) 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:476: set_chunk oid 27 offset: 185344 length: 419328 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 185344 snap -1 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:477: set_chunk oid 28 offset: 308224 length: 950784 target oid smithi15938319-28 low_tier offset: 308224 snap -1 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:477 retry set_chunk ! 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 15 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:471: finishing set_chunk to oid 22 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 999424 length: 108032 2024-09-13T13:50:14.664 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:470: finishing set_chunk to oid 21 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1299456 length: 187392 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:472: finishing set_chunk to oid 23 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 468480 length: 321536 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:467: finishing set_chunk to oid 18 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1420288 length: 148992 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:473: finishing set_chunk to oid 24 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 181760 length: 1433600 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:474: finishing set_chunk to oid 25 2024-09-13T13:50:14.665 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1378304 length: 523776 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:467: done (14 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:470: done (13 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:471: done (12 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:472: done (11 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:473: done (10 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:474: done (9 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:475: finishing set_chunk to oid 26 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1567744 length: 78848 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout:475: done (8 left) 2024-09-13T13:50:14.666 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:477: finishing set_chunk to oid 28 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 308224 length: 950784 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:477: done (7 left) 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:476: finishing set_chunk to oid 27 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 185344 length: 419328 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:476: done (6 left) 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:450: finishing set_chunk to oid 1 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout:450: done (5 left) 2024-09-13T13:50:14.667 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:469: finishing set_chunk to oid 20 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:469: done (4 left) 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:462: finishing set_chunk to oid 13 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:462: done (3 left) 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:448: finishing set_chunk to oid 49 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 3 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:448: done (2 left) 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:428: finishing set_chunk to oid 29 2024-09-13T13:50:14.668 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:428: done (1 left) 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:437: finishing set_chunk to oid 38 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:437: done (0 left) 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:478: set_chunk oid 29 offset: 274432 length: 2028032 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 274432 snap -1 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:479: set_chunk oid 30 offset: 846848 length: 172544 target oid smithi15938319-30 low_tier offset: 846848 snap -1 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:480: set_chunk oid 31 offset: 1702912 length: 493568 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1702912 snap -1 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:481: set_chunk oid 32 offset: 972288 length: 1537536 target oid smithi15938319-32 low_tier offset: 972288 snap -1 2024-09-13T13:50:14.669 INFO:tasks.rados.rados.0.smithi159.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:482: set_chunk oid 33 offset: 1735168 length: 522752 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1735168 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:483: set_chunk oid 34 offset: 409088 length: 1392640 target oid smithi15938319-34 low_tier offset: 409088 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:484: set_chunk oid 35 offset: 321024 length: 1370112 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 321024 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:485: set_chunk oid 36 offset: 797696 length: 1112576 target oid smithi15938319-36 low_tier offset: 797696 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:486: set_chunk oid 37 offset: 1015296 length: 421376 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1015296 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:487: set_chunk oid 38 offset: 17408 length: 1300992 target oid smithi15938319-38 low_tier offset: 17408 snap -1 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.670 INFO:tasks.rados.rados.0.smithi159.stdout:488: set_chunk oid 39 offset: 799744 length: 162816 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 799744 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:489: set_chunk oid 40 offset: 998912 length: 105472 target oid smithi15938319-40 low_tier offset: 998912 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:490: set_chunk oid 41 offset: 723968 length: 1793536 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 723968 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:491: set_chunk oid 42 offset: 395264 length: 2178048 target oid smithi15938319-42 low_tier offset: 395264 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:492: set_chunk oid 43 offset: 36864 length: 832000 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 36864 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:493: set_chunk oid 44 offset: 1100288 length: 1089024 target oid smithi15938319-44 low_tier offset: 1100288 snap -1 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:481: finishing set_chunk to oid 32 2024-09-13T13:50:14.671 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 972288 length: 1537536 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:483: finishing set_chunk to oid 34 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 409088 length: 1392640 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:481: done (15 left) 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:483: done (14 left) 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:494: set_chunk oid 45 offset: 806400 length: 272384 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 806400 snap -1 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:495: set_chunk oid 46 offset: 516096 length: 2019328 target oid smithi15938319-46 low_tier offset: 516096 snap -1 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:482: finishing set_chunk to oid 33 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1735168 length: 522752 2024-09-13T13:50:14.672 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:479: finishing set_chunk to oid 30 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 846848 length: 172544 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:486: finishing set_chunk to oid 37 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1015296 length: 421376 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:478: finishing set_chunk to oid 29 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 274432 length: 2028032 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:478: done (15 left) 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:479: done (14 left) 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:482: done (13 left) 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:486: done (12 left) 2024-09-13T13:50:14.673 INFO:tasks.rados.rados.0.smithi159.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:496: set_chunk oid 47 offset: 1845248 length: 858112 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1845248 snap -1 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:497: set_chunk oid 48 offset: 2399744 length: 346624 target oid smithi15938319-48 low_tier offset: 2399744 snap -1 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:498: set_chunk oid 49 offset: 1533952 length: 149504 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1533952 snap -1 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:485: finishing set_chunk to oid 36 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 797696 length: 1112576 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:491: finishing set_chunk to oid 42 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 395264 length: 2178048 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:490: finishing set_chunk to oid 41 2024-09-13T13:50:14.674 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 723968 length: 1793536 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:480: finishing set_chunk to oid 31 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1702912 length: 493568 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:480: done (14 left) 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:485: done (13 left) 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:490: done (12 left) 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:491: done (11 left) 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:499: set_chunk oid 50 offset: 2548736 length: 187904 target oid smithi15938319-50 low_tier offset: 2548736 snap -1 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:500: set_chunk oid 1 offset: 1175552 length: 690688 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1175552 snap -1 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.675 INFO:tasks.rados.rados.0.smithi159.stdout:501: set_chunk oid 2 offset: 384000 length: 1353216 target oid smithi15938319-2 low_tier offset: 384000 snap -1 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:484: finishing set_chunk to oid 35 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 321024 length: 1370112 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:487: finishing set_chunk to oid 38 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 17408 length: 1300992 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:494: finishing set_chunk to oid 45 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 806400 length: 272384 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:484: done (13 left) 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:487: done (12 left) 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:494: done (11 left) 2024-09-13T13:50:14.676 INFO:tasks.rados.rados.0.smithi159.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:502: set_chunk oid 3 offset: 906240 length: 20480 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 906240 snap -1 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:503: set_chunk oid 4 offset: 457728 length: 306176 target oid smithi15938319-4 low_tier offset: 457728 snap -1 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:504: set_chunk oid 5 offset: 160768 length: 1399808 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 160768 snap -1 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:505: set_chunk oid 6 offset: 193024 length: 1820672 target oid smithi15938319-6 low_tier offset: 193024 snap -1 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:506: set_chunk oid 7 offset: 800768 length: 479232 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 800768 snap -1 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:488: finishing set_chunk to oid 39 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 799744 length: 162816 2024-09-13T13:50:14.677 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:493: finishing set_chunk to oid 44 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1100288 length: 1089024 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:495: finishing set_chunk to oid 46 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 516096 length: 2019328 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:497: finishing set_chunk to oid 48 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 2399744 length: 346624 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:488: done (15 left) 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:493: done (14 left) 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:495: done (13 left) 2024-09-13T13:50:14.678 INFO:tasks.rados.rados.0.smithi159.stdout:497: done (12 left) 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:507: set_chunk oid 8 offset: 1594368 length: 476672 target oid smithi15938319-8 low_tier offset: 1594368 snap -1 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:508: set_chunk oid 9 offset: 202752 length: 1562624 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 202752 snap -1 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:509: set_chunk oid 10 offset: 192512 length: 1058304 target oid smithi15938319-10 low_tier offset: 192512 snap -1 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:510: set_chunk oid 11 offset: 449024 length: 779264 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 449024 snap -1 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:489: finishing set_chunk to oid 40 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 998912 length: 105472 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:498: finishing set_chunk to oid 49 2024-09-13T13:50:14.679 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1533952 length: 149504 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:496: finishing set_chunk to oid 47 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1845248 length: 858112 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:489: done (15 left) 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:496: done (14 left) 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:498: done (13 left) 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:511: set_chunk oid 12 offset: 403456 length: 1405952 target oid smithi15938319-12 low_tier offset: 403456 snap -1 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:512: set_chunk oid 13 offset: 641536 length: 35328 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 641536 snap -1 2024-09-13T13:50:14.680 INFO:tasks.rados.rados.0.smithi159.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:513: set_chunk oid 14 offset: 414720 length: 1021952 target oid smithi15938319-14 low_tier offset: 414720 snap -1 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:501: finishing set_chunk to oid 2 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 384000 length: 1353216 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:500: finishing set_chunk to oid 1 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1175552 length: 690688 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:500: done (15 left) 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:501: done (14 left) 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:514: set_chunk oid 15 offset: 1174528 length: 57344 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1174528 snap -1 2024-09-13T13:50:14.681 INFO:tasks.rados.rados.0.smithi159.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:515: set_chunk oid 16 offset: 522752 length: 549888 target oid smithi15938319-16 low_tier offset: 522752 snap -1 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:502: finishing set_chunk to oid 3 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 906240 length: 20480 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:502: done (15 left) 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:516: set_chunk oid 17 offset: 165888 length: 932864 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 165888 snap -1 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:504: finishing set_chunk to oid 5 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 160768 length: 1399808 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.682 INFO:tasks.rados.rados.0.smithi159.stdout:505: finishing set_chunk to oid 6 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 193024 length: 1820672 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:507: finishing set_chunk to oid 8 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1594368 length: 476672 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:510: finishing set_chunk to oid 11 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 449024 length: 779264 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:509: finishing set_chunk to oid 10 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 192512 length: 1058304 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:504: done (15 left) 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:505: done (14 left) 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:507: done (13 left) 2024-09-13T13:50:14.683 INFO:tasks.rados.rados.0.smithi159.stdout:509: done (12 left) 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:510: done (11 left) 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:517: set_chunk oid 18 offset: 924160 length: 147968 target oid smithi15938319-18 low_tier offset: 924160 snap -1 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:518: set_chunk oid 19 offset: 933376 length: 525824 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 933376 snap -1 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:519: set_chunk oid 20 offset: 1228288 length: 238592 target oid smithi15938319-20 low_tier offset: 1228288 snap -1 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:520: set_chunk oid 21 offset: 360448 length: 1575936 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 360448 snap -1 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:521: set_chunk oid 22 offset: 325120 length: 558592 target oid smithi15938319-22 low_tier offset: 325120 snap -1 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.684 INFO:tasks.rados.rados.0.smithi159.stdout:503: finishing set_chunk to oid 4 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 457728 length: 306176 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:508: finishing set_chunk to oid 9 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 202752 length: 1562624 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:512: finishing set_chunk to oid 13 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 641536 length: 35328 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:513: finishing set_chunk to oid 14 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 414720 length: 1021952 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:511: finishing set_chunk to oid 12 2024-09-13T13:50:14.685 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 403456 length: 1405952 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:503: done (15 left) 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:508: done (14 left) 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:511: done (13 left) 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:512: done (12 left) 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:513: done (11 left) 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:522: set_chunk oid 23 offset: 1118720 length: 967680 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1118720 snap -1 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:523: set_chunk oid 24 offset: 112640 length: 1712128 target oid smithi15938319-24 low_tier offset: 112640 snap -1 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:524: set_chunk oid 25 offset: 393216 length: 1263104 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 393216 snap -1 2024-09-13T13:50:14.686 INFO:tasks.rados.rados.0.smithi159.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:525: set_chunk oid 26 offset: 277504 length: 1921536 target oid smithi15938319-26 low_tier offset: 277504 snap -1 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:526: set_chunk oid 27 offset: 706560 length: 1026560 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 706560 snap -1 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:515: finishing set_chunk to oid 16 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 522752 length: 549888 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:514: finishing set_chunk to oid 15 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1174528 length: 57344 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:506: finishing set_chunk to oid 7 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 800768 length: 479232 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:520: finishing set_chunk to oid 21 2024-09-13T13:50:14.687 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 360448 length: 1575936 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:518: finishing set_chunk to oid 19 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 933376 length: 525824 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:517: finishing set_chunk to oid 18 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 924160 length: 147968 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:506: done (15 left) 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:514: done (14 left) 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:515: done (13 left) 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:517: done (12 left) 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:518: done (11 left) 2024-09-13T13:50:14.688 INFO:tasks.rados.rados.0.smithi159.stdout:520: done (10 left) 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:521: finishing set_chunk to oid 22 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 325120 length: 558592 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:519: finishing set_chunk to oid 20 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1228288 length: 238592 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:516: finishing set_chunk to oid 17 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 165888 length: 932864 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:527: set_chunk oid 28 offset: 477696 length: 1042944 target oid smithi15938319-28 low_tier offset: 477696 snap -1 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:516: done (10 left) 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:519: done (9 left) 2024-09-13T13:50:14.689 INFO:tasks.rados.rados.0.smithi159.stdout:521: done (8 left) 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:528: set_chunk oid 29 offset: 1593856 length: 328704 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1593856 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:529: set_chunk oid 30 offset: 532992 length: 1285120 target oid smithi15938319-30 low_tier offset: 532992 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:530: set_chunk oid 31 offset: 838144 length: 35328 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 838144 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:531: set_chunk oid 32 offset: 622592 length: 404992 target oid smithi15938319-32 low_tier offset: 622592 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:532: set_chunk oid 33 offset: 1348608 length: 404992 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1348608 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:533: set_chunk oid 34 offset: 1293824 length: 958464 target oid smithi15938319-34 low_tier offset: 1293824 snap -1 2024-09-13T13:50:14.690 INFO:tasks.rados.rados.0.smithi159.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:534: set_chunk oid 35 offset: 1453568 length: 857600 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1453568 snap -1 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:535: set_chunk oid 36 offset: 1782272 length: 713216 target oid smithi15938319-36 low_tier offset: 1782272 snap -1 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:522: finishing set_chunk to oid 23 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1118720 length: 967680 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:525: finishing set_chunk to oid 26 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 277504 length: 1921536 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:522: done (15 left) 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:525: done (14 left) 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:536: set_chunk oid 37 offset: 1641984 length: 267776 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1641984 snap -1 2024-09-13T13:50:14.691 INFO:tasks.rados.rados.0.smithi159.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:537: set_chunk oid 38 offset: 1180672 length: 169984 target oid smithi15938319-38 low_tier offset: 1180672 snap -1 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:523: finishing set_chunk to oid 24 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 112640 length: 1712128 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:526: finishing set_chunk to oid 27 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 706560 length: 1026560 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:523: done (15 left) 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:526: done (14 left) 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:538: set_chunk oid 39 offset: 1314304 length: 601600 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1314304 snap -1 2024-09-13T13:50:14.692 INFO:tasks.rados.rados.0.smithi159.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:539: set_chunk oid 40 offset: 2514944 length: 31232 target oid smithi15938319-40 low_tier offset: 2514944 snap -1 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:528: finishing set_chunk to oid 29 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1593856 length: 328704 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:527: finishing set_chunk to oid 28 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 477696 length: 1042944 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:530: finishing set_chunk to oid 31 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 838144 length: 35328 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.693 INFO:tasks.rados.rados.0.smithi159.stdout:533: finishing set_chunk to oid 34 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1293824 length: 958464 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:527: done (15 left) 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:528: done (14 left) 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:530: done (13 left) 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:533: done (12 left) 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:540: set_chunk oid 41 offset: 620032 length: 133632 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 620032 snap -1 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:532: finishing set_chunk to oid 33 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1348608 length: 404992 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:532: done (12 left) 2024-09-13T13:50:14.694 INFO:tasks.rados.rados.0.smithi159.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:541: set_chunk oid 42 offset: 1024 length: 2317312 target oid smithi15938319-42 low_tier offset: 1024 snap -1 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:541 retry set_chunk ! 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:534: finishing set_chunk to oid 35 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1453568 length: 857600 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:534: done (12 left) 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 12 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:531: finishing set_chunk to oid 32 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 622592 length: 404992 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:531: done (11 left) 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 11 2024-09-13T13:50:14.695 INFO:tasks.rados.rados.0.smithi159.stdout:529: finishing set_chunk to oid 30 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 532992 length: 1285120 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:538: finishing set_chunk to oid 39 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1314304 length: 601600 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:535: finishing set_chunk to oid 36 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1782272 length: 713216 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:529: done (10 left) 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:535: done (9 left) 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:538: done (8 left) 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.696 INFO:tasks.rados.rados.0.smithi159.stdout:540: finishing set_chunk to oid 41 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 620032 length: 133632 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:540: done (7 left) 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 7 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:539: finishing set_chunk to oid 40 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 2514944 length: 31232 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:539: done (6 left) 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:541: finishing set_chunk to oid 42 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1024 length: 2317312 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.697 INFO:tasks.rados.rados.0.smithi159.stdout:541: done (5 left) 2024-09-13T13:50:14.698 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.698 INFO:tasks.rados.rados.0.smithi159.stdout:492: finishing set_chunk to oid 43 2024-09-13T13:50:14.698 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.698 INFO:tasks.rados.rados.0.smithi159.stdout:492: done (4 left) 2024-09-13T13:50:14.698 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.698 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:14.514+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:14.707 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:14.700+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:499: finishing set_chunk to oid 50 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:499: done (3 left) 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:536: finishing set_chunk to oid 37 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout:536: done (2 left) 2024-09-13T13:50:14.713 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:537: finishing set_chunk to oid 38 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:537: done (1 left) 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:524: finishing set_chunk to oid 25 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 393216 length: 1263104 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:524: done (0 left) 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:542: set_chunk oid 43 offset: 469504 length: 455680 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 469504 snap -1 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:543: set_chunk oid 44 offset: 67584 length: 1143296 target oid smithi15938319-44 low_tier offset: 67584 snap -1 2024-09-13T13:50:14.714 INFO:tasks.rados.rados.0.smithi159.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:544: set_chunk oid 45 offset: 473600 length: 462848 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 473600 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:545: set_chunk oid 46 offset: 2443776 length: 32768 target oid smithi15938319-46 low_tier offset: 2443776 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:546: set_chunk oid 47 offset: 2417664 length: 16384 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2417664 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:547: set_chunk oid 48 offset: 68608 length: 932352 target oid smithi15938319-48 low_tier offset: 68608 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:548: set_chunk oid 49 offset: 561664 length: 880128 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 561664 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:549: set_chunk oid 50 offset: 1082368 length: 230400 target oid smithi15938319-50 low_tier offset: 1082368 snap -1 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.715 INFO:tasks.rados.rados.0.smithi159.stdout:550: set_chunk oid 1 offset: 965632 length: 962048 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 965632 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:551: set_chunk oid 2 offset: 773120 length: 98816 target oid smithi15938319-2 low_tier offset: 773120 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:552: set_chunk oid 3 offset: 1364480 length: 137728 target oid smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1364480 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:553: set_chunk oid 4 offset: 673280 length: 633856 target oid smithi15938319-4 low_tier offset: 673280 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:554: set_chunk oid 5 offset: 1094144 length: 476160 target oid smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1094144 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:555: set_chunk oid 6 offset: 1283072 length: 512 target oid smithi15938319-6 low_tier offset: 1283072 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:556: set_chunk oid 7 offset: 968192 length: 974848 target oid smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 968192 snap -1 2024-09-13T13:50:14.716 INFO:tasks.rados.rados.0.smithi159.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:557: set_chunk oid 8 offset: 38400 length: 1387008 target oid smithi15938319-8 low_tier offset: 38400 snap -1 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:544: finishing set_chunk to oid 45 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 473600 length: 462848 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:547: finishing set_chunk to oid 48 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 68608 length: 932352 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:544: done (15 left) 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:547: done (14 left) 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.717 INFO:tasks.rados.rados.0.smithi159.stdout:558: set_chunk oid 9 offset: 1413120 length: 620032 target oid smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1413120 snap -1 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:559: set_chunk oid 10 offset: 1052672 length: 535552 target oid smithi15938319-10 low_tier offset: 1052672 snap -1 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:548: finishing set_chunk to oid 49 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 561664 length: 880128 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:548: done (15 left) 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:560: set_chunk oid 11 offset: 376320 length: 826368 target oid smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 376320 snap -1 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:550: finishing set_chunk to oid 1 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 965632 length: 962048 2024-09-13T13:50:14.718 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:545: finishing set_chunk to oid 46 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 2443776 length: 32768 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:542: finishing set_chunk to oid 43 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 469504 length: 455680 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:549: finishing set_chunk to oid 50 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1082368 length: 230400 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:542: done (15 left) 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:545: done (14 left) 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:549: done (13 left) 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:550: done (12 left) 2024-09-13T13:50:14.719 INFO:tasks.rados.rados.0.smithi159.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:561: set_chunk oid 12 offset: 1346048 length: 756224 target oid smithi15938319-12 low_tier offset: 1346048 snap -1 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:562: set_chunk oid 13 offset: 18944 length: 312320 target oid smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 18944 snap -1 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:563: set_chunk oid 14 offset: 151552 length: 108544 target oid smithi15938319-14 low_tier offset: 151552 snap -1 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:564: set_chunk oid 15 offset: 83968 length: 1322496 target oid smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 83968 snap -1 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:543: finishing set_chunk to oid 44 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 67584 length: 1143296 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:555: finishing set_chunk to oid 6 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1283072 length: 512 2024-09-13T13:50:14.720 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:546: finishing set_chunk to oid 47 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2417664 length: 16384 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:543: done (15 left) 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:546: done (14 left) 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:555: done (13 left) 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:565: set_chunk oid 16 offset: 1043456 length: 250880 target oid smithi15938319-16 low_tier offset: 1043456 snap -1 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:566: set_chunk oid 17 offset: 9728 length: 142336 target oid smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 9728 snap -1 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout:567: set_chunk oid 18 offset: 1690624 length: 267264 target oid smithi15938319-18 low_tier offset: 1690624 snap -1 2024-09-13T13:50:14.721 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:557: finishing set_chunk to oid 8 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 38400 length: 1387008 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:553: finishing set_chunk to oid 4 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 673280 length: 633856 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:553: done (15 left) 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:557: done (14 left) 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:568: set_chunk oid 19 offset: 1218048 length: 949760 target oid smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1218048 snap -1 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout:569: set_chunk oid 20 offset: 894976 length: 796160 target oid smithi15938319-20 low_tier offset: 894976 snap -1 2024-09-13T13:50:14.722 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:554: finishing set_chunk to oid 5 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1094144 length: 476160 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:558: finishing set_chunk to oid 9 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1413120 length: 620032 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:560: finishing set_chunk to oid 11 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 376320 length: 826368 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:554: done (15 left) 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:558: done (14 left) 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:560: done (13 left) 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.723 INFO:tasks.rados.rados.0.smithi159.stdout:570: set_chunk oid 21 offset: 1864192 length: 56832 target oid smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1864192 snap -1 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:571: set_chunk oid 22 offset: 668160 length: 1645568 target oid smithi15938319-22 low_tier offset: 668160 snap -1 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:572: set_chunk oid 23 offset: 2148352 length: 102912 target oid smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2148352 snap -1 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:561: finishing set_chunk to oid 12 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1346048 length: 756224 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:559: finishing set_chunk to oid 10 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1052672 length: 535552 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:556: finishing set_chunk to oid 7 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 968192 length: 974848 2024-09-13T13:50:14.724 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:563: finishing set_chunk to oid 14 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 151552 length: 108544 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:567: finishing set_chunk to oid 18 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1690624 length: 267264 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:562: finishing set_chunk to oid 13 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 18944 length: 312320 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:565: finishing set_chunk to oid 16 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1043456 length: 250880 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:14.725 INFO:tasks.rados.rados.0.smithi159.stdout:556: done (15 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:559: done (14 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:561: done (13 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:562: done (12 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:563: done (11 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:565: done (10 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:567: done (9 left) 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:573: set_chunk oid 24 offset: 1966592 length: 20992 target oid smithi15938319-24 low_tier offset: 1966592 snap -1 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:574: set_chunk oid 25 offset: 720384 length: 1471488 target oid smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 720384 snap -1 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:575: set_chunk oid 26 offset: 1024 length: 568832 target oid smithi15938319-26 low_tier offset: 1024 snap -1 2024-09-13T13:50:14.726 INFO:tasks.rados.rados.0.smithi159.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:576: set_chunk oid 27 offset: 2049024 length: 77824 target oid smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2049024 snap -1 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:577: set_chunk oid 28 offset: 1170944 length: 880640 target oid smithi15938319-28 low_tier offset: 1170944 snap -1 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:578: set_chunk oid 29 offset: 238592 length: 754176 target oid smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 238592 snap -1 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:579: set_chunk oid 30 offset: 1004032 length: 22528 target oid smithi15938319-30 low_tier offset: 1004032 snap -1 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:564: finishing set_chunk to oid 15 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 83968 length: 1322496 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:568: finishing set_chunk to oid 19 2024-09-13T13:50:14.727 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1218048 length: 949760 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:566: finishing set_chunk to oid 17 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 9728 length: 142336 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:569: finishing set_chunk to oid 20 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 894976 length: 796160 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:564: done (15 left) 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:566: done (14 left) 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:568: done (13 left) 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:569: done (12 left) 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:580: set_chunk oid 31 offset: 1995776 length: 246784 target oid smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1995776 snap -1 2024-09-13T13:50:14.728 INFO:tasks.rados.rados.0.smithi159.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:581: set_chunk oid 32 offset: 717824 length: 581632 target oid smithi15938319-32 low_tier offset: 717824 snap -1 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:582: set_chunk oid 33 offset: 731136 length: 1654272 target oid smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 731136 snap -1 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:583: set_chunk oid 34 offset: 328192 length: 433664 target oid smithi15938319-34 low_tier offset: 328192 snap -1 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:570: finishing set_chunk to oid 21 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1864192 length: 56832 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:571: finishing set_chunk to oid 22 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 668160 length: 1645568 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T13:50:14.729 INFO:tasks.rados.rados.0.smithi159.stdout:570: done (15 left) 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:571: done (14 left) 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:584: set_chunk oid 35 offset: 748544 length: 326144 target oid smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 748544 snap -1 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:585: set_chunk oid 36 offset: 341504 length: 373760 target oid smithi15938319-36 low_tier offset: 341504 snap -1 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:574: finishing set_chunk to oid 25 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 720384 length: 1471488 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:575: finishing set_chunk to oid 26 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1024 length: 568832 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:577: finishing set_chunk to oid 28 2024-09-13T13:50:14.730 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1170944 length: 880640 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:574: done (15 left) 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:575: done (14 left) 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:577: done (13 left) 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:586: set_chunk oid 37 offset: 157696 length: 2232320 target oid smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 157696 snap -1 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:587: set_chunk oid 38 offset: 671232 length: 200192 target oid smithi15938319-38 low_tier offset: 671232 snap -1 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:588: set_chunk oid 39 offset: 323072 length: 1315840 target oid smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 323072 snap -1 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:573: finishing set_chunk to oid 24 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1966592 length: 20992 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T13:50:14.731 INFO:tasks.rados.rados.0.smithi159.stdout:580: finishing set_chunk to oid 31 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1995776 length: 246784 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:573: done (15 left) 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:580: done (14 left) 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:589: set_chunk oid 40 offset: 8704 length: 1324544 target oid smithi15938319-40 low_tier offset: 8704 snap -1 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:576: finishing set_chunk to oid 27 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2049024 length: 77824 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:581: finishing set_chunk to oid 32 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 717824 length: 581632 2024-09-13T13:50:14.732 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:590: set_chunk oid 41 offset: 986624 length: 738304 target oid smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 986624 snap -1 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:576: done (15 left) 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:581: done (14 left) 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:591: set_chunk oid 42 offset: 1276416 length: 714752 target oid smithi15938319-42 low_tier offset: 1276416 snap -1 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:592: set_chunk oid 43 offset: 104448 length: 1450496 target oid smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 104448 snap -1 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:578: finishing set_chunk to oid 29 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 238592 length: 754176 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:583: finishing set_chunk to oid 34 2024-09-13T13:50:14.733 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 328192 length: 433664 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:582: finishing set_chunk to oid 33 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 731136 length: 1654272 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:578: done (15 left) 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:582: done (14 left) 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:583: done (13 left) 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:593: set_chunk oid 44 offset: 129024 length: 1834496 target oid smithi15938319-44 low_tier offset: 129024 snap -1 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:594: set_chunk oid 45 offset: 1017344 length: 223232 target oid smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1017344 snap -1 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:595: set_chunk oid 46 offset: 973312 length: 566784 target oid smithi15938319-46 low_tier offset: 973312 snap -1 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:14.734 INFO:tasks.rados.rados.0.smithi159.stdout:588: finishing set_chunk to oid 39 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 323072 length: 1315840 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:589: finishing set_chunk to oid 40 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 8704 length: 1324544 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:588: done (15 left) 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:589: done (14 left) 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:596: set_chunk oid 47 offset: 129536 length: 388608 target oid smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 129536 snap -1 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:587: finishing set_chunk to oid 38 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 671232 length: 200192 2024-09-13T13:50:14.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:586: finishing set_chunk to oid 37 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 157696 length: 2232320 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:590: finishing set_chunk to oid 41 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 986624 length: 738304 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:586: done (14 left) 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:587: done (13 left) 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:590: done (12 left) 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:597: set_chunk oid 48 offset: 112128 length: 2068992 target oid smithi15938319-48 low_tier offset: 112128 snap -1 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.736 INFO:tasks.rados.rados.0.smithi159.stdout:598: set_chunk oid 49 offset: 799232 length: 1132032 target oid smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 799232 snap -1 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:591: finishing set_chunk to oid 42 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1276416 length: 714752 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:593: finishing set_chunk to oid 44 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 129024 length: 1834496 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:592: finishing set_chunk to oid 43 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 104448 length: 1450496 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:591: done (13 left) 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:592: done (12 left) 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:593: done (11 left) 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-09-13T13:50:14.737 INFO:tasks.rados.rados.0.smithi159.stdout:599: set_chunk oid 50 offset: 102400 length: 314880 target oid smithi15938319-50 low_tier offset: 102400 snap -1 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:600: set_chunk oid 1 offset: 1492480 length: 239616 target oid smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1492480 snap -1 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:601 oid_not_in_use 37 oid_redirect_not_in_use 0 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:600 wait for manifest initialization 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 13 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:594: finishing set_chunk to oid 45 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1017344 length: 223232 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:595: finishing set_chunk to oid 46 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 973312 length: 566784 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:596: finishing set_chunk to oid 47 2024-09-13T13:50:14.738 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 129536 length: 388608 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:594: done (12 left) 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:595: done (11 left) 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:596: done (10 left) 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 10 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:597: finishing set_chunk to oid 48 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 112128 length: 2068992 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:597: done (9 left) 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 9 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:598: finishing set_chunk to oid 49 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 799232 length: 1132032 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:14.739 INFO:tasks.rados.rados.0.smithi159.stdout:598: done (8 left) 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 8 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:600: finishing set_chunk to oid 1 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1492480 length: 239616 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:599: finishing set_chunk to oid 50 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 102400 length: 314880 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:599: done (7 left) 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:600: done (6 left) 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 6 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:551: finishing set_chunk to oid 2 2024-09-13T13:50:14.740 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:551: done (5 left) 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:552: finishing set_chunk to oid 3 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:552: done (4 left) 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:585: finishing set_chunk to oid 36 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:585: done (3 left) 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 3 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:584: finishing set_chunk to oid 35 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T13:50:14.741 INFO:tasks.rados.rados.0.smithi159.stdout:584: done (2 left) 2024-09-13T13:50:14.742 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 2 2024-09-13T13:50:14.742 INFO:tasks.rados.rados.0.smithi159.stdout:579: finishing set_chunk to oid 30 2024-09-13T13:50:14.742 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T13:50:14.742 INFO:tasks.rados.rados.0.smithi159.stdout:579: done (1 left) 2024-09-13T13:50:14.742 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:50:14.743 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:14.743 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:14.743 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:14.802 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:14.801+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:14.846 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:14.847 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:14.855 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:14.902 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:14.902+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:14.958 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:14.958 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:14.966 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:15.004 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:15.003+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:15.069 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.069 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout:572: finishing set_chunk to oid 23 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout:572: done (0 left) 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 52 low_tier 2024-09-13T13:50:15.099 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 54 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 56 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 58 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 60 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 62 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 64 low_tier 2024-09-13T13:50:15.100 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 66 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 68 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 70 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 72 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 74 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 76 low_tier 2024-09-13T13:50:15.101 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 78 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 80 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 82 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 84 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 86 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 88 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 90 low_tier 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.102 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 92 low_tier 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 94 low_tier 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 96 low_tier 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 98 low_tier 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout: redirect_not_in_use: 100 low_tier 2024-09-13T13:50:15.103 INFO:tasks.rados.rados.0.smithi159.stdout:601: set_chunk oid 43 target oid 2024-09-13T13:50:15.104 INFO:tasks.rados.rados.0.smithi159.stdout:601: set_chunk oid 43 offset: 1778176 length: 440832 target oid 3b80f8209607cc6e23496e0c542629951ce6a7218aaafd41a5b18ac09d13b2f6 offset: 0 snap -1 2024-09-13T13:50:15.105 INFO:tasks.rados.rados.0.smithi159.stdout:602: read oid 29 snap -1 2024-09-13T13:50:15.105 INFO:tasks.rados.rados.0.smithi159.stdout:602: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T13:50:15.105 INFO:tasks.rados.rados.0.smithi159.stdout:603: write (excl) oid 22 current snap is 0 2024-09-13T13:50:15.105 INFO:tasks.rados.rados.0.smithi159.stdout:603: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-13T13:50:15.105 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:15.103+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:15.108 INFO:tasks.rados.rados.0.smithi159.stdout:603: writing smithi15938319-22 from 440350 to 900918 tid 1 2024-09-13T13:50:15.114 INFO:tasks.rados.rados.0.smithi159.stdout:603: writing smithi15938319-22 from 1309629 to 1852295 tid 2 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:603: writing smithi15938319-22 from 2323973 to 2840350 tid 3 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:604: read oid 26 snap -1 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:604: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:605: write oid 30 current snap is 0 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:601: finishing set_chunk to oid 43 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 1778176 length: 440832 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T13:50:15.120 INFO:tasks.rados.rados.0.smithi159.stdout:605: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-13T13:50:15.125 INFO:tasks.rados.rados.0.smithi159.stdout:605: writing smithi15938319-30 from 457157 to 992974 tid 1 2024-09-13T13:50:15.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:50:15.132 INFO:tasks.rados.rados.0.smithi159.stdout:605: writing smithi15938319-30 from 1768111 to 2170788 tid 2 2024-09-13T13:50:15.132 INFO:tasks.rados.rados.0.smithi159.stdout:601: done (4 left) 2024-09-13T13:50:15.132 INFO:tasks.rados.rados.0.smithi159.stdout:606: read oid 33 snap -1 2024-09-13T13:50:15.132 INFO:tasks.rados.rados.0.smithi159.stdout:606: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:15.132 INFO:tasks.rados.rados.0.smithi159.stdout:607: write oid 43 current snap is 0 2024-09-13T13:50:15.133 INFO:tasks.rados.rados.0.smithi159.stdout:602: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T13:50:15.152 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:15.186 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.186 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.190 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1694292 2024-09-13T13:50:15.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:15.198 INFO:tasks.rados.rados.0.smithi159.stdout:603: finishing write tid 1 to smithi15938319-22 2024-09-13T13:50:15.198 INFO:tasks.rados.rados.0.smithi159.stdout:603: finishing write tid 2 to smithi15938319-22 2024-09-13T13:50:15.198 INFO:tasks.rados.rados.0.smithi159.stdout:603: finishing write tid 3 to smithi15938319-22 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: finishing write tid 4 to smithi15938319-22 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: finishing write tid 5 to smithi15938319-22 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: oid 22 updating version 0 to 9 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: oid 22 version 9 is already newer than 6 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: oid 22 version 9 is already newer than 7 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: oid 22 version 9 is already newer than 8 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:603: left oid 22 (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:15.199 INFO:tasks.rados.rados.0.smithi159.stdout:604: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T13:50:15.205 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:15.205+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:15.226 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:15.253 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:15.281 INFO:tasks.rados.rados.0.smithi159.stdout:605: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:15.282 INFO:tasks.rados.rados.0.smithi159.stdout:605: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:15.282 INFO:tasks.rados.rados.0.smithi159.stdout:607: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-13T13:50:15.287 INFO:tasks.rados.rados.0.smithi159.stdout:607: writing smithi15938319-43 from 473964 to 1085030 tid 1 2024-09-13T13:50:15.294 INFO:tasks.rados.rados.0.smithi159.stdout:607: writing smithi15938319-43 from 1826593 to 2372928 tid 2 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:607: writing smithi15938319-43 from 2778819 to 2873964 tid 3 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:602: done (5 left) 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:603: done (4 left) 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:604: done (3 left) 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:608: snap_create 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:605: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:605: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:605: oid 30 updating version 0 to 29 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:605: oid 30 updating version 29 to 31 2024-09-13T13:50:15.298 INFO:tasks.rados.rados.0.smithi159.stdout:605: oid 30 version 31 is already newer than 30 2024-09-13T13:50:15.299 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 31 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-13T13:50:15.299 INFO:tasks.rados.rados.0.smithi159.stdout:605: left oid 30 (ObjNum 51 snap 0 seq_num 51) 2024-09-13T13:50:15.299 INFO:tasks.rados.rados.0.smithi159.stdout:606: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:15.300 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1351292 2024-09-13T13:50:15.303 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.303 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.306 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:15.305+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:15.312 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:15.318 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 253378 2024-09-13T13:50:15.371 INFO:tasks.rados.rados.0.smithi159.stdout:607: finishing write tid 1 to smithi15938319-43 2024-09-13T13:50:15.371 INFO:tasks.rados.rados.0.smithi159.stdout:607: finishing write tid 2 to smithi15938319-43 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: finishing write tid 3 to smithi15938319-43 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: finishing write tid 4 to smithi15938319-43 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: finishing write tid 5 to smithi15938319-43 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: oid 43 updating version 0 to 22 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: oid 43 updating version 22 to 24 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: oid 43 updating version 24 to 25 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: oid 43 version 25 is already newer than 23 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 25 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T13:50:15.372 INFO:tasks.rados.rados.0.smithi159.stdout:607: left oid 43 (ObjNum 52 snap 0 seq_num 52) 2024-09-13T13:50:15.406 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:15.406+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:15.425 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.425 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.431 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:15.507 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:15.506+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:15.533 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.533 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:15.608 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:15.607+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:15.644 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.644 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.653 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:15.710 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:15.709+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:15.754 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.755 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.763 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:15.810 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:15.810+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:15.865 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.865 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.873 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:15.911 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:15.909+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:15.985 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:15.985 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:15.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:16.011 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:16.010+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:16.101 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.101 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:16.112 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:16.111+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:16.118 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:16.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:16.118 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T13:50:16.118 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:50:16.118 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:50:16.118 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-09-13T13:50:16.119 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:50:16.123 INFO:tasks.rados.rados.0.smithi159.stdout:605: done (3 left) 2024-09-13T13:50:16.123 INFO:tasks.rados.rados.0.smithi159.stdout:606: done (2 left) 2024-09-13T13:50:16.123 INFO:tasks.rados.rados.0.smithi159.stdout:607: done (1 left) 2024-09-13T13:50:16.123 INFO:tasks.rados.rados.0.smithi159.stdout:608: done (0 left) 2024-09-13T13:50:16.123 INFO:tasks.rados.rados.0.smithi159.stdout:609: snap_create 2024-09-13T13:50:16.214 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:16.213+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:16.218 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.218 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:16.230 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:16.230 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:50:16.230 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:16.315 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:16.314+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:16.334 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.334 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:16.415 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:16.414+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:16.451 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.451 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:16.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:16.517 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:16.516+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:16.608 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.609 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:16.618 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:16.617+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:16.679 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:16.680 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:16.702 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:16.718 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:16.717+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:16.723 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:16.723 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:16.727 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.727 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:16.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:16.766 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:16.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:16.787 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:16.788 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:16.819 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:16.818+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:16.843 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:16.843 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:16.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:16.906 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:16.906 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:16.919 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:16.919+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:16.928 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:16.928 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:16.949 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:16.949 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:16.970 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:16.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:16.992 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:16.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:17.002 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:17.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.8350543459999997, 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:17.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.008 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.8350283530000002, 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:50:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:19 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.8300828320000004, 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096180398, 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170583+0000", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170585+0000", 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1090000000000001e-06 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266590+0000", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096005323000000004 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266598+0000", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8110000000000003e-06 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266650+0000", 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2481000000000001e-05 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266658+0000", 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266763+0000", 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105209 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.7297195309999998, 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:50:17.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:50:17.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.022 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:111 4.2 e41/34 4:4d382039:::smithi15938319-28:head v 41'9, mlcod=41'5)", 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5566796319999998, 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.06900017, 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1600000000000001e-07 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443991+0000", 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512563+0000", 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.068572151999999997 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512567+0000", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7119999999999999e-06 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512607+0000", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249000000000003e-05 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512623+0000", 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5866999999999999e-05 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512961+0000", 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033803199999999999 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512986+0000", 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5079000000000002e-05 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.027 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:135 4.a 4:5004af91:::smithi15938319-34:head [setxattr _header (56) in=7b,truncate 2554631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4867587909999997, 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034151982999999997, 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 135 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513907+0000", 2024-09-13T13:50:17.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.99e-07 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513910+0000", 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2210000000000002e-06 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513922+0000", 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1858999999999999e-05 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513929+0000", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1069999999999999e-06 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514737+0000", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080765899999999996 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514809+0000", 2024-09-13T13:50:17.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1928999999999995e-05 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514899+0000", 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0511000000000001e-05 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515444+0000", 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054460299999999995 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547991+0000", 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032546974999999999 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548022+0000", 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1442999999999997e-05 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548058+0000", 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6077e-05 2024-09-13T13:50:17.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:136 4.a 4:5004af91:::smithi15938319-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4867177959999998, 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034245790999999998, 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 136 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513948+0000", 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0200000000000002e-07 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513951+0000", 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2909999999999999e-06 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513958+0000", 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7950000000000002e-06 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513964+0000", 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.874e-06 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515064+0000", 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001099865 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515106+0000", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2398000000000003e-05 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548087+0000", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032980559 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548133+0000", 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6343e-05 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548193+0000", 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0064e-05 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:135 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4849765289999999, 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032374238999999999, 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.039 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8099999999999999e-07 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515690+0000", 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7000000000000003e-07 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547939+0000", 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032248700999999998 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547943+0000", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6990000000000002e-06 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547981+0000", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7302999999999998e-05 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547985+0000", 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000002e-06 2024-09-13T13:50:17.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548063+0000", 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7894999999999999e-05 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:149 4.a 4:5c84836d:::smithi15938319-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4424845050000004, 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011167603, 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.043 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 149 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558182+0000", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1799999999999999e-06 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558629+0000", 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044714800000000002 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558640+0000", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0631999999999999e-05 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558645+0000", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0379999999999999e-06 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558697+0000", 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1931000000000001e-05 2024-09-13T13:50:17.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558781+0000", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4166999999999996e-05 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558864+0000", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2762999999999993e-05 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.559564+0000", 2024-09-13T13:50:17.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00070006799999999998 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569279+0000", 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097149720000000005 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569319+0000", 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0296000000000002e-05 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569348+0000", 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9408e-05 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4336843540000004, 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:50:17.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4336698539999997, 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.055 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:50:17.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:17.060 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.4314164140000001, 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:17.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.064 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.3089931259999998, 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:50:17.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.070 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.3089623069999998, 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:50:17.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:50:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.3068041990000001, 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.079 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:209 3.3 e41/33 3:c3255dc7:::smithi15938319-10 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.2433512609999999, 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012424223, 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757315+0000", 2024-09-13T13:50:17.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6799999999999999e-07 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759545+0000", 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002230793 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759568+0000", 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2237000000000001e-05 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759574+0000", 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.618e-06 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759615+0000", 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1078999999999998e-05 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759631+0000", 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5914e-05 2024-09-13T13:50:17.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.769699+0000", 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010068027 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.769738+0000", 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8787e-05 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:236 3.3 e41/33 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'10, mlcod=41'6)", 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.1935935610000001, 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012626577999999999, 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5999999999999999e-07 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810513+0000", 2024-09-13T13:50:17.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034412800000000001 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810543+0000", 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0054999999999999e-05 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810552+0000", 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810615+0000", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2788999999999998e-05 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810643+0000", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7535999999999999e-05 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819621+0000", 2024-09-13T13:50:17.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089780380000000007 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819698+0000", 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7553999999999996e-05 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.0972700010000001, 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:50:17.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.231292469, 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:50:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.2176167549999999, 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:50:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:50:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.103 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:17.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:17.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:17.106 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:17.106 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:17.106 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:17.076+0000 7f2b25d9b640 0 -- 172.21.15.27:0/3566527077 >> v2:172.21.15.122:6808/1255553857 conn(0x7f2b28221d70 msgr2=0x7f2b28224160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:17.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:17.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:17.107 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:17.108 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:17.103+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:609: done (0 left) 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:610: read oid 27 snap -1 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:610: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:611: read oid 46 snap -1 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:611: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:612: tier_promote oid 3 2024-09-13T13:50:17.134 INFO:tasks.rados.rados.0.smithi159.stdout:613: delete oid 44 current snap is 2 2024-09-13T13:50:17.137 INFO:tasks.rados.rados.0.smithi159.stdout:610: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T13:50:17.201 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2212259 2024-09-13T13:50:17.203 INFO:tasks.rados.rados.0.smithi159.stdout:611: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:17.205 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:17.204+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:17.217 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:17.217 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:17.217 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:17.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:17.235 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1262680 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:610: done (3 left) 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:611: done (2 left) 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:613: done (1 left) 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:614: read oid 38 snap -1 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:614: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:17.280 INFO:tasks.rados.rados.0.smithi159.stdout:615: snap_create 2024-09-13T13:50:17.285 INFO:tasks.rados.rados.0.smithi159.stdout:614: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:17.305 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:17.304+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:17.317 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1748388 2024-09-13T13:50:17.332 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:17.333 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:17.333 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:17.333 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:17.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:17.407 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:17.406+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.452 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:21 4.d 4:b771c1ad:::smithi15938319-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2745082659999998, 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010356456999999999, 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:17.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175385+0000", 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.238e-06 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176269+0000", 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088463099999999998 2024-09-13T13:50:17.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176280+0000", 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0831e-05 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176289+0000", 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176354+0000", 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5102000000000001e-05 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176360+0000", 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0929999999999999e-06 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180157+0000", 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003796982 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180282+0000", 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012503899999999999 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180445+0000", 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016250300000000001 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184519+0000", 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040741090000000002 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185660+0000", 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001141186 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185696+0000", 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6143000000000003e-05 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185740+0000", 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3634000000000003e-05 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:22 4.d 4:b771c1ad:::smithi15938319-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2735873699999996, 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097627319999999997, 2024-09-13T13:50:17.461 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 22 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176305+0000", 2024-09-13T13:50:17.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-07 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176334+0000", 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8750000000000001e-05 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176341+0000", 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1350000000000003e-06 2024-09-13T13:50:17.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176347+0000", 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6110000000000001e-06 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176428+0000", 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1902000000000003e-05 2024-09-13T13:50:17.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176431+0000", 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8590000000000001e-06 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181669+0000", 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052375959999999997 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181723+0000", 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4209999999999998e-05 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181816+0000", 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2446000000000002e-05 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184783+0000", 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029669269999999998 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185959+0000", 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011760799999999999 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186016+0000", 2024-09-13T13:50:17.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7726999999999998e-05 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186067+0000", 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0812000000000001e-05 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.469 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:23 4.d 4:b771c1ad:::smithi15938319-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2735341709999997, 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098262599999999999, 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 23 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176362+0000", 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4750000000000002e-06 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176368+0000", 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176373+0000", 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1279999999999999e-06 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176463+0000", 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9957999999999995e-05 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176469+0000", 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.925e-06 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182113+0000", 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056435290000000004 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182187+0000", 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3993999999999997e-05 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:17.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182323+0000", 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136122 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184790+0000", 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024670999999999998 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186114+0000", 2024-09-13T13:50:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001323765 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186142+0000", 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8067000000000001e-05 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186184+0000", 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1922000000000003e-05 2024-09-13T13:50:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:24 4.d 4:b771c1ad:::smithi15938319-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2735082609999999, 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010036734, 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9100000000000004e-07 2024-09-13T13:50:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176387+0000", 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6060000000000001e-06 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176409+0000", 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2161e-05 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176417+0000", 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2309999999999996e-06 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176484+0000", 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7453999999999995e-05 2024-09-13T13:50:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176486+0000", 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.95e-06 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182553+0000", 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060667480000000003 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182596+0000", 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2447000000000002e-05 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186229+0000", 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036334179999999998 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186308+0000", 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8979000000000004e-05 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186420+0000", 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112249 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:46 4.c 4:34c00adb:::smithi15938319-12:head [write 1610596~562003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2047484380000002, 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051361310000000004, 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 46 2024-09-13T13:50:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245144+0000", 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2600000000000001e-07 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245643+0000", 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049847100000000001 2024-09-13T13:50:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245654+0000", 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1473e-05 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245660+0000", 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5450000000000003e-06 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245716+0000", 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5908000000000001e-05 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245803+0000", 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7570000000000004e-05 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245941+0000", 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013756599999999999 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247677+0000", 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0017366440000000001 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250179+0000", 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025012039999999999 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250235+0000", 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6073000000000003e-05 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250279+0000", 2024-09-13T13:50:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4750999999999999e-05 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:47 4.c 4:34c00adb:::smithi15938319-12:head [setxattr _header (56) in=7b,truncate 2184877] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.245672+0000", 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2042202050000004, 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0047553029999999998, 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 47 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245672+0000", 2024-09-13T13:50:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245672+0000", 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245672+0000", 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1399999999999997e-07 2024-09-13T13:50:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245674+0000", 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.215e-06 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245678+0000", 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7110000000000001e-06 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245682+0000", 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1269999999999996e-06 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.246837+0000", 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011546760000000001 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.246937+0000", 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000100076 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247078+0000", 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014110899999999999 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247714+0000", 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063571000000000005 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250349+0000", 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026354939999999999 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250380+0000", 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1170999999999997e-05 2024-09-13T13:50:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250427+0000", 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6499999999999999e-05 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:48 4.c 4:34c00adb:::smithi15938319-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.245706+0000", 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2041861340000004, 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049374279999999998, 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 48 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245706+0000", 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245706+0000", 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245706+0000", 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1800000000000002e-07 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245708+0000", 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.601e-06 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245712+0000", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0879999999999997e-06 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245715+0000", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2990000000000001e-06 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247305+0000", 2024-09-13T13:50:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015896110000000001 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247349+0000", 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4029000000000001e-05 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250471+0000", 2024-09-13T13:50:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031221280000000001 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250544+0000", 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3318999999999994e-05 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250643+0000", 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9035000000000003e-05 2024-09-13T13:50:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:174 4.c 4:34327e96:::smithi15938319-44:head [write 1816608~788001] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.7965391640000004, 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005761937, 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 174 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653354+0000", 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0359999999999999e-06 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654276+0000", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092214800000000002 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654286+0000", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0251000000000001e-05 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654298+0000", 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1576999999999999e-05 2024-09-13T13:50:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654371+0000", 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3044000000000001e-05 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654420+0000", 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9453000000000002e-05 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654488+0000", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7547999999999998e-05 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655731+0000", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012433909999999999 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659062+0000", 2024-09-13T13:50:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033305679999999999 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659091+0000", 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9448999999999999e-05 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659115+0000", 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3472000000000002e-05 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:175 4.c 4:34327e96:::smithi15938319-44:head [setxattr _header (56) in=7b,truncate 2722701] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.654313+0000", 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.7955787369999996, 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049103599999999999, 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 175 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654313+0000", 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654313+0000", 2024-09-13T13:50:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654314+0000", 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654317+0000", 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.134e-06 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654324+0000", 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.686e-06 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654330+0000", 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4400000000000002e-06 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655059+0000", 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072862100000000004 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655107+0000", 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8928999999999999e-05 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655188+0000", 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0174999999999997e-05 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655796+0000", 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060792599999999995 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659148+0000", 2024-09-13T13:50:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033521660000000002 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659180+0000", 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1819000000000001e-05 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659223+0000", 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3825000000000001e-05 2024-09-13T13:50:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:176 4.c 4:34327e96:::smithi15938319-44:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.654372+0000", 2024-09-13T13:50:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.7955200050000002, 2024-09-13T13:50:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049718059999999996, 2024-09-13T13:50:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 176 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654372+0000", 2024-09-13T13:50:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654372+0000", 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654372+0000", 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:50:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654376+0000", 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5149999999999998e-06 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654383+0000", 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4889999999999998e-06 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654389+0000", 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2989999999999996e-06 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655344+0000", 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095491 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655387+0000", 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3232000000000002e-05 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659248+0000", 2024-09-13T13:50:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038607469999999999 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659289+0000", 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1032999999999998e-05 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659344+0000", 2024-09-13T13:50:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5031000000000002e-05 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.14.23:9 4.c 4:34327e96:::smithi15938319-44:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817214+0000", 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.6326774940000002, 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050514490000000004, 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817214+0000", 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817214+0000", 2024-09-13T13:50:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817216+0000", 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9250000000000002e-06 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817224+0000", 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.712e-06 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817254+0000", 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0034000000000001e-05 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817262+0000", 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6909999999999997e-06 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817319+0000", 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7606999999999999e-05 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817368+0000", 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8934000000000003e-05 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822266+0000", 2024-09-13T13:50:17.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004897546 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.3.22:13 4.c 4:358f784f:::smithi15938319-41:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.6321318600000003, 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092837100000000006, 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817761+0000", 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4529999999999999e-06 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817768+0000", 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.764e-06 2024-09-13T13:50:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817788+0000", 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9417000000000001e-05 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817795+0000", 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4619999999999999e-06 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822308+0000", 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045129460000000003 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822421+0000", 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011317800000000001 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.827044+0000", 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046224899999999999 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5517381280000002, 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:50:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:50:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5516702709999999, 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:50:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:50:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.5383873460000004, 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:50:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:50:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:311 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.6802146840000001, 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091613149999999997, 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 311 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769678+0000", 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.19e-06 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769687+0000", 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5210000000000002e-06 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769699+0000", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2346e-05 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769708+0000", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769790+0000", 2024-09-13T13:50:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2278000000000001e-05 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769891+0000", 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101059 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777673+0000", 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077819710000000004 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778488+0000", 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00081508000000000004 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778779+0000", 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029088900000000001 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778807+0000", 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8079e-05 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778838+0000", 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1411000000000001e-05 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:355 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074384+0000", 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.375508269, 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005030695, 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 355 2024-09-13T13:50:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074384+0000", 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074384+0000", 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074386+0000", 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2520000000000002e-06 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074396+0000", 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0617e-05 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074429+0000", 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2527000000000002e-05 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074442+0000", 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2642e-05 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074529+0000", 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7272000000000001e-05 2024-09-13T13:50:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074623+0000", 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3868000000000003e-05 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077748+0000", 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031254429999999999 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.078358+0000", 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00061016399999999995 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079329+0000", 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00097073400000000001 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079374+0000", 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5311e-05 2024-09-13T13:50:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079414+0000", 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9864999999999999e-05 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.375110678, 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:50:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:50:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:390 4.c 4:358f784f:::smithi15938319-41:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 3.3702844679999999, 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065600609999999998, 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 390 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079609+0000", 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.438e-06 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079614+0000", 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1000000000000003e-06 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079626+0000", 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2361000000000001e-05 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079633+0000", 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079679+0000", 2024-09-13T13:50:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6201e-05 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079730+0000", 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1313999999999997e-05 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085125+0000", 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053946200000000001 2024-09-13T13:50:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085782+0000", 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00065719300000000003 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086055+0000", 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027300700000000002 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086118+0000", 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2491999999999997e-05 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086167+0000", 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9811999999999997e-05 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:786 4.c 4:34327e96:::smithi15938319-44:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e44)", 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.31570679600000001, 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006240097, 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 786 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134187+0000", 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.193e-06 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134195+0000", 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2660000000000001e-06 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134213+0000", 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7999000000000002e-05 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134223+0000", 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2129999999999997e-06 2024-09-13T13:50:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134283+0000", 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0652999999999997e-05 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134289+0000", 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4330000000000003e-06 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.136814+0000", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002525158 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.136917+0000", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102877 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.137114+0000", 2024-09-13T13:50:17.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019731499999999999 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.139063+0000", 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019493749999999999 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140320+0000", 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012565199999999999 2024-09-13T13:50:17.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140374+0000", 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4017999999999997e-05 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140425+0000", 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1076999999999999e-05 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:17.599 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:17.599 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:17.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:17.601 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:17.599+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:17.701 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:17.700+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:17.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:17.802 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:17.801+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:17.828 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:17.828 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:17.828 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:17.829 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:17.829 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:17.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:17.903 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:17.902+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1 4.7 4:e63fe140:::smithi15938319-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8220161670000001, 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044289289999999999, 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117795+0000", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8100000000000001e-07 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118374+0000", 2024-09-13T13:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00057889599999999997 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118388+0000", 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3969e-05 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118399+0000", 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1168999999999999e-05 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118711+0000", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00031199899999999998 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118800+0000", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8660999999999997e-05 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118899+0000", 2024-09-13T13:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9193999999999999e-05 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120073+0000", 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001174476 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122149+0000", 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020752969999999998 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122184+0000", 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5830000000000001e-05 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122223+0000", 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8457000000000001e-05 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2 4.7 4:e63fe140:::smithi15938319-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.119435+0000", 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8203753369999998, 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029055249999999999, 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.119435+0000", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.119435+0000", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.119436+0000", 2024-09-13T13:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8700000000000004e-07 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120104+0000", 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066844599999999995 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120114+0000", 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0158e-05 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120123+0000", 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.568e-06 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120175+0000", 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1711e-05 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120253+0000", 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8568999999999999e-05 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120376+0000", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123087 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121495+0000", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011184439999999999 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122282+0000", 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078756099999999999 2024-09-13T13:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122310+0000", 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999000000000001e-05 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122340+0000", 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9995e-05 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3 4.7 4:e63fe140:::smithi15938319-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8196702729999998, 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045172090000000003, 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120141+0000", 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0699999999999996e-07 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120144+0000", 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.331e-06 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120150+0000", 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1109999999999996e-06 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120155+0000", 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9370000000000003e-06 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121069+0000", 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00091422599999999997 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121156+0000", 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6473999999999999e-05 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121286+0000", 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000130662 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121691+0000", 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040453800000000002 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124593+0000", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029016990000000002 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124624+0000", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1055e-05 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124657+0000", 2024-09-13T13:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3368999999999998e-05 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4 4.7 4:e63fe140:::smithi15938319-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.819646348, 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058803479999999997, 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4499999999999997e-07 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120167+0000", 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8700000000000001e-06 2024-09-13T13:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120173+0000", 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2269999999999998e-06 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120182+0000", 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4819999999999994e-06 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121521+0000", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001339668 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121596+0000", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4987000000000006e-05 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125923+0000", 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043266620000000002 2024-09-13T13:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125983+0000", 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300000000000002e-05 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.126044+0000", 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0707e-05 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:73 4.7 4:e1bd516b:::smithi15938319-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6140792150000003, 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060681499999999996, 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 73 2024-09-13T13:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325734+0000", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326420+0000", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068555600000000003 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326435+0000", 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.575e-05 2024-09-13T13:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326444+0000", 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8410000000000006e-06 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326517+0000", 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2784e-05 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326630+0000", 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112759 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326729+0000", 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9761999999999998e-05 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.328491+0000", 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00176133 2024-09-13T13:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331735+0000", 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003244317 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331768+0000", 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2750000000000003e-05 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331799+0000", 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1281000000000001e-05 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:74 4.7 4:e1bd516b:::smithi15938319-19:head [write 1903617~398909] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6097823040000003, 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041941720000000004, 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 74 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330029+0000", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.217e-06 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330323+0000", 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029364499999999998 2024-09-13T13:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330334+0000", 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0903999999999999e-05 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330340+0000", 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330413+0000", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3460999999999993e-05 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330483+0000", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9626e-05 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330556+0000", 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3460000000000005e-05 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331402+0000", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00084570499999999998 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334176+0000", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027738569999999998 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334202+0000", 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5732999999999999e-05 2024-09-13T13:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334222+0000", 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0465000000000001e-05 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:75 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6094572620000003, 2024-09-13T13:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040696860000000003, 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 75 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330374+0000", 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0729000000000001e-05 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330389+0000", 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4990999999999999e-05 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330398+0000", 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.397e-06 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330944+0000", 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054573899999999995 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331015+0000", 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0405000000000001e-05 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331096+0000", 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1872000000000005e-05 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331592+0000", 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00049587600000000002 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334365+0000", 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027724540000000002 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334396+0000", 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1449000000000003e-05 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334423+0000", 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6302e-05 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:76 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6093907920000001, 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041363490000000001, 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 76 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330420+0000", 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330422+0000", 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5210000000000001e-06 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330429+0000", 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3090000000000001e-06 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330434+0000", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0300000000000001e-06 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331244+0000", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081012399999999996 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331316+0000", 2024-09-13T13:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2653999999999999e-05 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334451+0000", 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031348529999999999 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334493+0000", 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1644000000000003e-05 2024-09-13T13:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334556+0000", 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2714999999999998e-05 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:97 4.7 4:e26a40dd:::smithi15938319-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.397953+0000", 2024-09-13T13:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5418572560000001, 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026098470000000002, 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 97 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.397953+0000", 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.397953+0000", 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.397956+0000", 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6599999999999999e-06 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398307+0000", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035182799999999999 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398322+0000", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4916000000000001e-05 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398329+0000", 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5169999999999996e-06 2024-09-13T13:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398400+0000", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1278000000000004e-05 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398491+0000", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0692000000000005e-05 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.398582+0000", 2024-09-13T13:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1448999999999998e-05 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.399560+0000", 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00097813999999999996 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.400492+0000", 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093212700000000004 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.400524+0000", 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1124000000000002e-05 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.400563+0000", 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9116000000000003e-05 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:98 4.7 4:e26a40dd:::smithi15938319-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5352965530000002, 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052413879999999996, 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 98 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2999999999999999e-07 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405146+0000", 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063201000000000002 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405157+0000", 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0773e-05 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405162+0000", 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2719999999999997e-06 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405203+0000", 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0639000000000002e-05 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405254+0000", 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0760000000000002e-05 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405333+0000", 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9501999999999999e-05 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406334+0000", 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010006590000000001 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409690+0000", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033561060000000002 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409720+0000", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9771999999999999e-05 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409755+0000", 2024-09-13T13:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5064999999999997e-05 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:99 4.7 4:e26a40dd:::smithi15938319-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5346357269999995, 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046796340000000002, 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 99 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405175+0000", 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405179+0000", 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.653e-06 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405184+0000", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.575e-06 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405188+0000", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7500000000000001e-06 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405851+0000", 2024-09-13T13:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066292300000000005 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405905+0000", 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4012999999999999e-05 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405988+0000", 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3009000000000004e-05 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406517+0000", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052932299999999995 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409789+0000", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032720230000000002 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409821+0000", 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1550000000000001e-05 2024-09-13T13:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409854+0000", 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3327999999999997e-05 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:100 4.7 4:e26a40dd:::smithi15938319-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5346147800000001, 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047806309999999996, 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 100 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405196+0000", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1100000000000001e-07 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405198+0000", 2024-09-13T13:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9709999999999998e-06 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405203+0000", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6629999999999998e-06 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405208+0000", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6149999999999999e-06 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406140+0000", 2024-09-13T13:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093174499999999999 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406184+0000", 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4008999999999998e-05 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409882+0000", 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036979159999999999 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409921+0000", 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9019e-05 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409976+0000", 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5281999999999997e-05 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.13.21:4 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.776166+0000", 2024-09-13T13:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.1636445740000001, 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015908859999999999, 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776166+0000", 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776166+0000", 2024-09-13T13:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776167+0000", 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-06 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776172+0000", 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776183+0000", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1539000000000001e-05 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776189+0000", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7180000000000001e-06 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776230+0000", 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0657000000000001e-05 2024-09-13T13:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.776276+0000", 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6213e-05 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.777756+0000", 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0014805230000000001 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.10.22:12 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.903630+0000", 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0361798469999997, 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001864896, 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903630+0000", 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903630+0000", 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903632+0000", 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2780000000000001e-06 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903636+0000", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5379999999999996e-06 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903647+0000", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0859999999999999e-05 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903655+0000", 2024-09-13T13:50:18.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9359999999999999e-06 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903698+0000", 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2713000000000003e-05 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903746+0000", 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8053000000000002e-05 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.905495+0000", 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001749518 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0245195149999997, 2024-09-13T13:50:18.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:50:18.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:50:18.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0197199860000001, 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.032 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:50:18.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:50:18.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:50:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.1703719479999997, 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:50:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:50:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:318 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.770378+0000", 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.1694317510000003, 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030893169999999998, 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 318 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770378+0000", 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770378+0000", 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770379+0000", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4200000000000005e-07 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770390+0000", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0379e-05 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770399+0000", 2024-09-13T13:50:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2779999999999999e-06 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770405+0000", 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4999999999999996e-06 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770451+0000", 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6066000000000002e-05 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770526+0000", 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4721000000000005e-05 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.772317+0000", 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017912640000000001 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.772980+0000", 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066260699999999995 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773402+0000", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042238200000000001 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773441+0000", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8157000000000001e-05 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773468+0000", 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7121e-05 2024-09-13T13:50:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:324 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.770908+0000", 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.1689017709999998, 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027425190000000001, 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 324 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770908+0000", 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770908+0000", 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770909+0000", 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2899999999999999e-07 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770912+0000", 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8279999999999999e-06 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770918+0000", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0410000000000003e-06 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770921+0000", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.770958+0000", 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7066000000000001e-05 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.771031+0000", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3013e-05 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.772791+0000", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017599219999999999 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773344+0000", 2024-09-13T13:50:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055232100000000004 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773596+0000", 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025205600000000001 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773625+0000", 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9405000000000001e-05 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.773651+0000", 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5772999999999999e-05 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.8655716369999999, 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:50:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:50:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.063 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:18.065 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:18.064+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:18.070 INFO:tasks.rados.rados.0.smithi159.stdout:612: got (0) Success 2024-09-13T13:50:18.071 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:18.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:50:18.141 INFO:tasks.rados.rados.0.smithi159.stdout:612: done (2 left) 2024-09-13T13:50:18.141 INFO:tasks.rados.rados.0.smithi159.stdout:614: done (1 left) 2024-09-13T13:50:18.141 INFO:tasks.rados.rados.0.smithi159.stdout:615: done (0 left) 2024-09-13T13:50:18.141 INFO:tasks.rados.rados.0.smithi159.stdout:616: write (excl) oid 39 current snap is 3 2024-09-13T13:50:18.141 INFO:tasks.rados.rados.0.smithi159.stdout:616: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-13T13:50:18.146 INFO:tasks.rados.rados.0.smithi159.stdout:616: writing smithi15938319-39 from 507578 to 985495 tid 1 2024-09-13T13:50:18.147 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:18.146+0000 7f1a526ee640 -1 osd.13 44 handle_osd_map injecting map crc failure 2024-09-13T13:50:18.152 INFO:tasks.rados.rados.0.smithi159.stdout:616: writing smithi15938319-39 from 1543557 to 2093561 tid 2 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:616: writing smithi15938319-39 from 2833665 to 2907578 tid 3 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:617: read oid 14 snap -1 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:617: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:618: read oid 37 snap -1 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:618: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:619: write (excl) oid 19 current snap is 3 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:616: finishing write tid 1 to smithi15938319-39 2024-09-13T13:50:18.157 INFO:tasks.rados.rados.0.smithi159.stdout:619: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-13T13:50:18.162 INFO:tasks.rados.rados.0.smithi159.stdout:619: writing smithi15938319-19 from 524385 to 1077551 tid 1 2024-09-13T13:50:18.166 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:18.165+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:18.168 INFO:tasks.rados.rados.0.smithi159.stdout:619: writing smithi15938319-19 from 1602039 to 2012054 tid 2 2024-09-13T13:50:18.174 INFO:tasks.rados.rados.0.smithi159.stdout:619: writing smithi15938319-19 from 2577441 to 2924385 tid 3 2024-09-13T13:50:18.174 INFO:tasks.rados.rados.0.smithi159.stdout:620: write (excl) oid 24 current snap is 3 2024-09-13T13:50:18.174 INFO:tasks.rados.rados.0.smithi159.stdout:620: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-13T13:50:18.180 INFO:tasks.rados.rados.0.smithi159.stdout:620: writing smithi15938319-24 from 541192 to 1169607 tid 1 2024-09-13T13:50:18.185 INFO:tasks.rados.rados.0.smithi159.stdout:620: writing smithi15938319-24 from 1660521 to 2214194 tid 2 2024-09-13T13:50:18.190 INFO:tasks.rados.rados.0.smithi159.stdout:620: writing smithi15938319-24 from 2888511 to 2941192 tid 3 2024-09-13T13:50:18.190 INFO:tasks.rados.rados.0.smithi159.stdout:621: read oid 22 snap -1 2024-09-13T13:50:18.190 INFO:tasks.rados.rados.0.smithi159.stdout:621: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:18.190 INFO:tasks.rados.rados.0.smithi159.stdout:622: set_chunk oid 26 target oid 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:622: set_chunk oid 26 offset: 2347520 length: 67584 target oid 2e1d1f24ad4c0336267e3f21f9e4083cfc299a1a41b208c12f42e166ce33f631 offset: 0 snap -1 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:623: read oid 44 snap -1 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:623: expect deleted 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:616: finishing write tid 2 to smithi15938319-39 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:616: finishing write tid 3 to smithi15938319-39 2024-09-13T13:50:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:616: finishing write tid 4 to smithi15938319-39 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: finishing write tid 5 to smithi15938319-39 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: oid 39 updating version 0 to 26 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: oid 39 updating version 26 to 27 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: oid 39 version 27 is already newer than 25 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: oid 39 version 27 is already newer than 24 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 27 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:616: left oid 39 (ObjNum 54 snap 3 seq_num 54) 2024-09-13T13:50:18.199 INFO:tasks.rados.rados.0.smithi159.stdout:617: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T13:50:18.202 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 893044 2024-09-13T13:50:18.214 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.215 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:18.215 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:18.215 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.223 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:18.251 INFO:tasks.rados.rados.0.smithi159.stdout:619: finishing write tid 1 to smithi15938319-19 2024-09-13T13:50:18.252 INFO:tasks.rados.rados.0.smithi159.stdout:618: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-13T13:50:18.255 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 817341 2024-09-13T13:50:18.262 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:18.260+0000 7ffaeece8640 -1 osd.0 44 handle_osd_map injecting map crc failure 2024-09-13T13:50:18.266 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:18.265+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: finishing write tid 2 to smithi15938319-19 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: finishing write tid 3 to smithi15938319-19 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: finishing write tid 4 to smithi15938319-19 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: finishing write tid 5 to smithi15938319-19 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: oid 19 updating version 0 to 18 2024-09-13T13:50:18.323 INFO:tasks.rados.rados.0.smithi159.stdout:619: oid 19 updating version 18 to 21 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:619: oid 19 version 21 is already newer than 19 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:619: oid 19 version 21 is already newer than 20 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 21 (ObjNum 55 snap 3 seq_num 55) dirty exists 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:619: left oid 19 (ObjNum 55 snap 3 seq_num 55) 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: finishing write tid 1 to smithi15938319-24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: finishing write tid 2 to smithi15938319-24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: finishing write tid 3 to smithi15938319-24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: finishing write tid 4 to smithi15938319-24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: finishing write tid 5 to smithi15938319-24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: oid 24 updating version 0 to 27 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: oid 24 version 27 is already newer than 26 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: oid 24 version 27 is already newer than 24 2024-09-13T13:50:18.324 INFO:tasks.rados.rados.0.smithi159.stdout:620: oid 24 version 27 is already newer than 25 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 27 (ObjNum 56 snap 3 seq_num 56) dirty exists 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:620: left oid 24 (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:624: read oid 34 snap -1 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:624: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:616: done (8 left) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:617: done (7 left) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:618: done (6 left) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:619: done (5 left) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:620: done (4 left) 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:625: delete oid 19 current snap is 3 2024-09-13T13:50:18.325 INFO:tasks.rados.rados.0.smithi159.stdout:621: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:18.350 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:18.365 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.365 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:18.365 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:18.366 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:18.366 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.367 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:18.366+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:18.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:18.422 INFO:tasks.rados.rados.0.smithi159.stdout:621: done (4 left) 2024-09-13T13:50:18.422 INFO:tasks.rados.rados.0.smithi159.stdout:623: done (3 left) 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:625: done (2 left) 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:626: read oid 42 snap -1 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:626: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:627: read oid 16 snap 2 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:627: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:628: tier_promote oid 28 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:622: finishing set_chunk to oid 26 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:622: done (4 left) 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:629: tier_promote oid 21 2024-09-13T13:50:18.423 INFO:tasks.rados.rados.0.smithi159.stdout:624: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T13:50:18.425 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1354264 2024-09-13T13:50:18.467 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:18.467+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:18.477 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1011662 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:29 4.4 e41/34 4:2fda4b5c:::smithi15938319-8:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:18.479 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.193233+0000", 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.2844968349999997, 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001770745, 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193233+0000", 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193233+0000", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193233+0000", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2100000000000002e-07 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193831+0000", 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059747200000000004 2024-09-13T13:50:18.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193854+0000", 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3257000000000001e-05 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193861+0000", 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.294e-06 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193923+0000", 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1759000000000003e-05 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.193943+0000", 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0336e-05 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.194969+0000", 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010255869999999999 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.195004+0000", 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4619000000000002e-05 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:30 4.4 e41/34 4:2fda4b5c:::smithi15938319-8:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.203129+0000", 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.274600317, 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031621380000000001, 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203129+0000", 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203129+0000", 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203130+0000", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3399999999999998e-07 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203228+0000", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7483000000000002e-05 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205397+0000", 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021692009999999999 2024-09-13T13:50:18.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205405+0000", 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.354e-06 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205452+0000", 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6755000000000002e-05 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205470+0000", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7833999999999999e-05 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.206251+0000", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078068799999999998 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.206292+0000", 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0988999999999997e-05 2024-09-13T13:50:18.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:31 4.4 e41/34 4:2fda4b5c:::smithi15938319-8:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.203234+0000", 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.274496064, 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003219951, 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.490 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203234+0000", 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203234+0000", 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203234+0000", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9900000000000001e-07 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.203237+0000", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0809999999999998e-06 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205427+0000", 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021893889999999999 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205432+0000", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5269999999999998e-06 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205843+0000", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00041121199999999999 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.205865+0000", 2024-09-13T13:50:18.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1367999999999999e-05 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.206417+0000", 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055271599999999997 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.206454+0000", 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6259e-05 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.494 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:81 4.4 e41/34 4:21c00763:::smithi15938319-21:head v 41'4, mlcod=41'2)", 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.350404+0000", 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.1273262380000002, 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021358639999999999, 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350404+0000", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350404+0000", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350405+0000", 2024-09-13T13:50:18.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1599999999999999e-06 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350831+0000", 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042674500000000002 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350861+0000", 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9652999999999999e-05 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350869+0000", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9049999999999997e-06 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350917+0000", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8513999999999997e-05 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.350932+0000", 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4961e-05 2024-09-13T13:50:18.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.352509+0000", 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001576346 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.352539+0000", 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:82 4.4 e41/34 4:21c00763:::smithi15938319-21:head v 41'5, mlcod=41'3)", 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.355858+0000", 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.1218721289999998, 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001097165, 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.355858+0000", 2024-09-13T13:50:18.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.355858+0000", 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.355859+0000", 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7630000000000001e-06 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356084+0000", 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000225064 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356139+0000", 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4583000000000002e-05 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356168+0000", 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9023e-05 2024-09-13T13:50:18.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356229+0000", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1146e-05 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356243+0000", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3304e-05 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356929+0000", 2024-09-13T13:50:18.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068623999999999996 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356955+0000", 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6041999999999999e-05 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:83 4.4 e41/34 4:21c00763:::smithi15938319-21:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.356528+0000", 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.1212017799999998, 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001322731, 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356528+0000", 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356528+0000", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356528+0000", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356536+0000", 2024-09-13T13:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7940000000000003e-06 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356554+0000", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7567999999999998e-05 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356559+0000", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.113e-06 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356593+0000", 2024-09-13T13:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3646000000000003e-05 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.356606+0000", 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3698e-05 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.357822+0000", 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001216033 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.357851+0000", 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8428999999999998e-05 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:85 4.9 e41/40 4:977d569f:::smithi15938319-22:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.367777+0000", 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.1099531389999999, 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024403189999999998, 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.367777+0000", 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.367777+0000", 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.367777+0000", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368778+0000", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001001196 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368796+0000", 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8139999999999999e-05 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368802+0000", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0209999999999996e-06 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368872+0000", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9584000000000003e-05 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368884+0000", 2024-09-13T13:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2248e-05 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370190+0000", 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013053030000000001 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370217+0000", 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7341000000000001e-05 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:86 4.9 e41/40 4:977d569f:::smithi15938319-22:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.368876+0000", 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.108854182, 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001548673, 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368876+0000", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368876+0000", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.368876+0000", 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369226+0000", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035021100000000001 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369244+0000", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7689e-05 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369250+0000", 2024-09-13T13:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5659999999999997e-06 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369288+0000", 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8247999999999997e-05 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369305+0000", 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.679e-05 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370406+0000", 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011013489999999999 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370424+0000", 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8321e-05 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:87 4.9 e41/40 4:977d569f:::smithi15938319-22:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.369333+0000", 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.108397235, 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001112018, 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369333+0000", 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369333+0000", 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369333+0000", 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1200000000000003e-07 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369338+0000", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1909999999999999e-06 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369357+0000", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.859e-05 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369362+0000", 2024-09-13T13:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1000000000000003e-06 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369767+0000", 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040480699999999998 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.369779+0000", 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1894e-05 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370431+0000", 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065286299999999995 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.370445+0000", 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3061000000000001e-05 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:165 4.4 e41/34 4:22f57a3c:::smithi15938319-42:head v 41'7, mlcod=41'5)", 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.615206+0000", 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8625233210000003, 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013513749999999999, 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615206+0000", 2024-09-13T13:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615206+0000", 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615208+0000", 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615541+0000", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000332944 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615565+0000", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4703999999999999e-05 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615573+0000", 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8220000000000007e-06 2024-09-13T13:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615640+0000", 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6973999999999999e-05 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.615656+0000", 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6098000000000001e-05 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.616524+0000", 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00086838799999999999 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.616558+0000", 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3309999999999998e-05 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:166 4.4 e41/34 4:22f57a3c:::smithi15938319-42:head v 41'8, mlcod=41'6)", 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.622316+0000", 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8554137900000001, 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001919844, 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.622316+0000", 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.622316+0000", 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.622316+0000", 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9700000000000002e-07 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623021+0000", 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00070498800000000003 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623039+0000", 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7873999999999999e-05 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623045+0000", 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4609999999999999e-06 2024-09-13T13:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623107+0000", 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2531000000000002e-05 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623121+0000", 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.366e-05 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.624208+0000", 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001086793 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.624236+0000", 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8140000000000002e-05 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:167 4.4 e41/34 4:22f57a3c:::smithi15938319-42:head v 41'9, mlcod=41'6)", 2024-09-13T13:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.623168+0000", 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8545622250000005, 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011055660000000001, 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623168+0000", 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623168+0000", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623168+0000", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623173+0000", 2024-09-13T13:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1329999999999998e-06 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623191+0000", 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7893e-05 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623196+0000", 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1989999999999997e-06 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623656+0000", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045995299999999998 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.623669+0000", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2542e-05 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.624252+0000", 2024-09-13T13:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00058292000000000001 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.624273+0000", 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1373e-05 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:193 4.0 4:04ca156e:::smithi15938319-49:head [write 406736~710070,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.716451+0000", 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.7612790770000002, 2024-09-13T13:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003622723, 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 193 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.716451+0000", 2024-09-13T13:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.716451+0000", 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.716453+0000", 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717199+0000", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074662999999999997 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717209+0000", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3789999999999995e-06 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717214+0000", 2024-09-13T13:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3319999999999999e-06 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717295+0000", 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0992e-05 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717398+0000", 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102897 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.717492+0000", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3904000000000001e-05 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718790+0000", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012983459999999999 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.719984+0000", 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011938610000000001 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720028+0000", 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4128999999999997e-05 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720073+0000", 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5187e-05 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:194 4.0 4:04ca156e:::smithi15938319-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.759664817, 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042375900000000003, 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 194 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000005e-07 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718779+0000", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071307200000000003 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718789+0000", 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0902e-05 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718798+0000", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8070000000000006e-06 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718838+0000", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9759999999999999e-05 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718890+0000", 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2086000000000002e-05 2024-09-13T13:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718968+0000", 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8289999999999998e-05 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720182+0000", 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001213492 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722239+0000", 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020566180000000001 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722277+0000", 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8420999999999997e-05 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722303+0000", 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5579e-05 2024-09-13T13:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:195 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (56) in=7b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.718812+0000", 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.7589177639999996, 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035967500000000001, 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 195 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718812+0000", 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718812+0000", 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718813+0000", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8500000000000001e-07 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718816+0000", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5240000000000001e-06 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718821+0000", 2024-09-13T13:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3720000000000004e-06 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718828+0000", 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7109999999999998e-06 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.719642+0000", 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000814272 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.719710+0000", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7985999999999996e-05 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.719821+0000", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110609 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720380+0000", 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055852600000000005 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722337+0000", 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019577930000000002 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722369+0000", 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1646000000000002e-05 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722409+0000", 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9725999999999999e-05 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:196 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.718837+0000", 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.7588930219999996, 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036960159999999999, 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 196 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718837+0000", 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718837+0000", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718837+0000", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8500000000000002e-07 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718839+0000", 2024-09-13T13:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718845+0000", 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718849+0000", 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.775e-06 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720110+0000", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001260277 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720159+0000", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9462999999999997e-05 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722435+0000", 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022760300000000001 2024-09-13T13:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722477+0000", 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1634000000000002e-05 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722533+0000", 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5934999999999999e-05 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.13.21:13 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.903146+0000", 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5745842359999997, 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022639040000000002, 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903146+0000", 2024-09-13T13:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903146+0000", 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903147+0000", 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.144e-06 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903157+0000", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0672999999999999e-05 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903168+0000", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1097999999999999e-05 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903178+0000", 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0028e-05 2024-09-13T13:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903225+0000", 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6610000000000003e-05 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.903273+0000", 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8194e-05 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.905409+0000", 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021361570000000001 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:271 4.9 e41/40 4:977d569f:::smithi15938319-22:head v 41'4, mlcod=41'2)", 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.930001+0000", 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5477283630000001, 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001210224, 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930001+0000", 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930001+0000", 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930003+0000", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.95e-06 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930036+0000", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2959000000000001e-05 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930099+0000", 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2259000000000001e-05 2024-09-13T13:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930115+0000", 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6801999999999999e-05 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930193+0000", 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7527000000000005e-05 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930214+0000", 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1361000000000001e-05 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931171+0000", 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00095672400000000005 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931212+0000", 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0642000000000002e-05 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7082947370000001, 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.6992643049999998, 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.584 INFO:tasks.rados.rados.0.smithi159.stdout:626: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T13:50:18.584 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 654741 2024-09-13T13:50:18.584 INFO:tasks.rados.rados.0.smithi159.stdout:627: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T13:50:18.584 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:18.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:18.585 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:18.582+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:18.642 INFO:tasks.rados.rados.0.smithi159.stdout:628: got (0) Success 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:624: done (4 left) 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:626: done (3 left) 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:627: done (2 left) 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:628: done (1 left) 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:630: read oid 35 snap 1 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:630: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:631: tier_evict oid 29 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:631: tier_evict oid 29 snap -1 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:632: tier_promote oid 33 2024-09-13T13:50:18.643 INFO:tasks.rados.rados.0.smithi159.stdout:633: read oid 5 snap -1 2024-09-13T13:50:18.644 INFO:tasks.rados.rados.0.smithi159.stdout:633: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:18.644 INFO:tasks.rados.rados.0.smithi159.stdout:634: read oid 49 snap 1 2024-09-13T13:50:18.644 INFO:tasks.rados.rados.0.smithi159.stdout:634: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:18.644 INFO:tasks.rados.rados.0.smithi159.stdout:635: write oid 44 current snap is 3 2024-09-13T13:50:18.644 INFO:tasks.rados.rados.0.smithi159.stdout:635: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-13T13:50:18.650 INFO:tasks.rados.rados.0.smithi159.stdout:635: writing smithi15938319-44 from 574806 to 1353719 tid 1 2024-09-13T13:50:18.657 INFO:tasks.rados.rados.0.smithi159.stdout:635: writing smithi15938319-44 from 2061132 to 2618474 tid 2 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:636: read oid 27 snap -1 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:636: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:637: read oid 26 snap -1 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:637: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:638: read oid 2 snap -1 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:638: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:639: read oid 48 snap -1 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:639: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:640: read oid 12 snap -1 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:640: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:18.658 INFO:tasks.rados.rados.0.smithi159.stdout:641: snap_remove snap 0 2024-09-13T13:50:18.685 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:18.684+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:18.786 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:18.785+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:18.839 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:18.887 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:18.885+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.716407815, 2024-09-13T13:50:18.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:50:18.955 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.708373669, 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:18.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:50:18.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:50:18.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:18.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:18.962 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:18.963 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:18.987 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:18.987+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:19.066 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:19.066 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:19.066 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:19.066 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:19.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:19.089 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:19.088+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:19.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:19.139 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:50:19.145 INFO:tasks.rados.rados.0.smithi159.stdout:641: done (12 left) 2024-09-13T13:50:19.146 INFO:tasks.rados.rados.0.smithi159.stdout:642: set_chunk oid 9 target oid 2024-09-13T13:50:19.162 INFO:tasks.rados.rados.0.smithi159.stdout:642: set_chunk oid 9 offset: 569344 length: 1232896 target oid ca3dee2fb6eb2dd311731f432d15387775ffad36d3bbd88079dd782c1d16d3ea offset: 0 snap -1 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:643: read oid 23 snap -1 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:643: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:644: tier_flush oid 37 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:644: tier_flush oid 37 snap -1 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:634: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:19.163 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.182 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:19.182 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:19.189 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:19.188+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:19.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:19.234 INFO:tasks.rados.rados.0.smithi159.stdout:633: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:19.234 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.267 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:19.289 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:19.289 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:19.290 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:19.288+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:19.290 INFO:tasks.rados.rados.0.smithi159.stdout:630: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:19.290 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.296 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:19.296 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:41 4.f 4:fec51f4e:::smithi15938319-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.0628019999999996, 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087181849999999998, 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 41 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231909+0000", 2024-09-13T13:50:19.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.231914+0000", 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1869999999999996e-06 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232499+0000", 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00058453499999999996 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232510+0000", 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0796e-05 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232519+0000", 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7389999999999994e-06 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232585+0000", 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5976000000000007e-05 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.232590+0000", 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.234923+0000", 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023328820000000001 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.235015+0000", 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2182999999999995e-05 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.235114+0000", 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8932999999999996e-05 2024-09-13T13:50:19.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240135+0000", 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050208220000000003 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240564+0000", 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000429694 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240597+0000", 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2839000000000002e-05 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240627+0000", 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0127000000000001e-05 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:42 4.f 4:fec51f4e:::smithi15938319-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.0615583690000001, 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010249345999999999, 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 42 2024-09-13T13:50:19.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233153+0000", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2399999999999998e-07 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233820+0000", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066637700000000001 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233829+0000", 2024-09-13T13:50:19.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7919999999999998e-06 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233836+0000", 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9600000000000003e-06 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233886+0000", 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0929e-05 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233891+0000", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4800000000000003e-06 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.238648+0000", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047567679999999998 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239101+0000", 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000452872 2024-09-13T13:50:19.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239171+0000", 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0440999999999999e-05 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240313+0000", 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011419170000000001 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243338+0000", 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030247899999999999 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243370+0000", 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2308999999999998e-05 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243402+0000", 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2187000000000002e-05 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.311 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:43 4.f 4:fec51f4e:::smithi15938319-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.0608610499999998, 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097138150000000006, 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 43 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233850+0000", 2024-09-13T13:50:19.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233851+0000", 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4499999999999997e-07 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233853+0000", 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2869999999999999e-06 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233862+0000", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0459999999999994e-06 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233866+0000", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0550000000000001e-06 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233913+0000", 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7360000000000001e-05 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233916+0000", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7070000000000001e-06 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239867+0000", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059507259999999999 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.239925+0000", 2024-09-13T13:50:19.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8322999999999997e-05 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240006+0000", 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0562000000000006e-05 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240882+0000", 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00087657100000000003 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243503+0000", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002620603 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243530+0000", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.673e-05 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243564+0000", 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.43e-05 2024-09-13T13:50:19.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:44 4.f 4:fec51f4e:::smithi15938319-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.0608275970000003, 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098119149999999992, 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:19.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 44 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233884+0000", 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3500000000000002e-07 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233887+0000", 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7020000000000001e-06 2024-09-13T13:50:19.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233890+0000", 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6669999999999998e-06 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233893+0000", 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1310000000000002e-06 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233935+0000", 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1909000000000001e-05 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.233939+0000", 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.721e-06 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240164+0000", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062245 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.240207+0000", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3377999999999997e-05 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243598+0000", 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033911689999999999 2024-09-13T13:50:19.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243637+0000", 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9276000000000001e-05 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.243696+0000", 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8127000000000001e-05 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5279077670000003, 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:50:19.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:50:19.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:50:19.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5269305690000001, 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:50:19.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5251759040000001, 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:50:19.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5247836909999997, 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.345 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:19.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:50:19.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.351 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5247463640000003, 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:50:19.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:50:19.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5199480110000003, 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.358 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:19.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5043127859999998, 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.364 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:19.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:50:19.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.5042064360000005, 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.370 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:50:19.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.4962726660000003, 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.377 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:50:19.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.478009922, 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:50:19.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:19.389 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.4778841079999996, 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:50:19.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:50:19.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.4777573180000001, 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.396 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:50:19.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:50:19.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3965588950000001, 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:19.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:50:19.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3964574440000002, 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.408 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:50:19.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:50:19.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:339 4.f 4:f883ee18:::smithi15938319-40:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.777405+0000", 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5173058020000001, 2024-09-13T13:50:19.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085288440000000007, 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 339 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777405+0000", 2024-09-13T13:50:19.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777405+0000", 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777406+0000", 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1500000000000004e-07 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777410+0000", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.941e-06 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777418+0000", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5279999999999998e-06 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777422+0000", 2024-09-13T13:50:19.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2880000000000003e-06 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777463+0000", 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1325999999999997e-05 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777505+0000", 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2043999999999999e-05 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.782312+0000", 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048064329999999997 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.782804+0000", 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00049231700000000004 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.785879+0000", 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003075061 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.785909+0000", 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0218000000000001e-05 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.785934+0000", 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4973e-05 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:791 4.f 4:fb8e36a2:::smithi15938319-39:head [write 507578~477917,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.146564+0000", 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.1481472930000001, 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077694230000000001, 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 791 2024-09-13T13:50:19.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.146564+0000", 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.146564+0000", 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.146566+0000", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-06 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.147118+0000", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055190000000000003 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.147135+0000", 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7906000000000001e-05 2024-09-13T13:50:19.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.147144+0000", 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.147213+0000", 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8516000000000006e-05 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:19.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.147221+0000", 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6629999999999999e-06 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.150415+0000", 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031937100000000002 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.150527+0000", 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112285 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.150753+0000", 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000225484 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.154037+0000", 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032848 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:19.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.154210+0000", 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017249999999999999 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.154273+0000", 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3394999999999999e-05 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.154333+0000", 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9982000000000003e-05 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.427 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:19.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:19.428 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:19.429 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:19.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:19.430 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 961769 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1231285 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:631: got (22) Invalid argument 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:632: got (0) Success 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:635: finishing write tid 1 to smithi15938319-44 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:637: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1157115 2024-09-13T13:50:19.431 INFO:tasks.rados.rados.0.smithi159.stdout:638: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T13:50:19.431 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:19.432 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:19.428+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:19.432 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:19.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:19.433 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:19.433 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:19.433 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:19.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:19.434 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:19.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:19.435 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:19.439 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:19.439 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:19.460 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:19.482 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:19.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:19.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:19.481+0000 7fd29bfff640 0 -- 172.21.15.27:0/972737045 >> v2:172.21.15.159:6809/1984230453 conn(0x7fd29c210db0 msgr2=0x7fd29c2131a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:50:19.488 INFO:tasks.rados.rados.0.smithi159.stdout:636: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-13T13:50:19.488 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.530 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:19.529+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:19.542 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1351477 2024-09-13T13:50:19.549 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:19.550 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:19.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:19.559 INFO:tasks.rados.rados.0.smithi159.stdout:635: finishing write tid 2 to smithi15938319-44 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: finishing write tid 3 to smithi15938319-44 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: finishing write tid 4 to smithi15938319-44 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: oid 44 updating version 0 to 25 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: oid 44 version 25 is already newer than 24 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: oid 44 version 25 is already newer than 23 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 25 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:635: left oid 44 (ObjNum 58 snap 3 seq_num 58) 2024-09-13T13:50:19.560 INFO:tasks.rados.rados.0.smithi159.stdout:639: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-13T13:50:19.603 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.631 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:19.629+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:19.634 INFO:tasks.rados.rados.0.smithi159.stdout:640: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:19.703 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:19.703 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:19.703 INFO:tasks.rados.rados.0.smithi159.stdout:629: got (0) Success 2024-09-13T13:50:19.703 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:19.704 INFO:tasks.rados.rados.0.smithi159.stdout:642: finishing set_chunk to oid 9 2024-09-13T13:50:19.704 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 569344 length: 1232896 2024-09-13T13:50:19.704 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:19.704 INFO:tasks.rados.rados.0.smithi159.stdout:643: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-13T13:50:19.704 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:19.705 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:19.705 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:19.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:19.725 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:19.732 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:19.730+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:19.747 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:19.748 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:19.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:19.747+0000 7fd29bfff640 0 -- 172.21.15.27:0/972737045 >> v2:172.21.15.122:6808/1255553857 conn(0x7fd29c21a6e0 msgr2=0x7fd29c21cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:629: done (14 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:630: done (13 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:631: done (12 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:632: done (11 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:633: done (10 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:634: done (9 left) 2024-09-13T13:50:19.783 INFO:tasks.rados.rados.0.smithi159.stdout:635: done (8 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:636: done (7 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:637: done (6 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:638: done (5 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:639: done (4 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:640: done (3 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:642: done (2 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:643: done (1 left) 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:645: write oid 14 current snap is 3 2024-09-13T13:50:19.784 INFO:tasks.rados.rados.0.smithi159.stdout:645: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-13T13:50:19.788 INFO:tasks.rados.rados.0.smithi159.stdout:645: writing smithi15938319-14 from 591613 to 1045775 tid 1 2024-09-13T13:50:19.793 INFO:tasks.rados.rados.0.smithi159.stdout:645: writing smithi15938319-14 from 1719614 to 2136967 tid 2 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:645: writing smithi15938319-14 from 2570780 to 2991613 tid 3 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:646: read oid 36 snap 1 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:646: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:647: read oid 12 snap -1 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:647: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:19.798 INFO:tasks.rados.rados.0.smithi159.stdout:648: set_chunk oid 49 target oid 2024-09-13T13:50:19.809 INFO:tasks.rados.rados.0.smithi159.stdout:648: set_chunk oid 49 offset: 793088 length: 884736 target oid 2c8466e1c73d7749037c4bdceee05b2ce4a4f395ceb8b7f2fa7fcced5be20977 offset: 0 snap -1 2024-09-13T13:50:19.810 INFO:tasks.rados.rados.0.smithi159.stdout:649: read oid 34 snap -1 2024-09-13T13:50:19.810 INFO:tasks.rados.rados.0.smithi159.stdout:649: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T13:50:19.810 INFO:tasks.rados.rados.0.smithi159.stdout:650: read oid 35 snap -1 2024-09-13T13:50:19.810 INFO:tasks.rados.rados.0.smithi159.stdout:650: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:19.810 INFO:tasks.rados.rados.0.smithi159.stdout:651: set_chunk oid 28 target oid 2024-09-13T13:50:19.816 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:19.817 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:19.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:19.833 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:19.832+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:19.851 INFO:tasks.rados.rados.0.smithi159.stdout:651: set_chunk oid 28 offset: 1350656 length: 880640 target oid cad54ae2783f462f40bbc5d7b65653983dcb535cd20426300c8fcb8819776881 offset: 0 snap -1 2024-09-13T13:50:19.851 INFO:tasks.rados.rados.0.smithi159.stdout:652: write (excl) oid 42 current snap is 3 2024-09-13T13:50:19.851 INFO:tasks.rados.rados.0.smithi159.stdout:652: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-13T13:50:19.857 INFO:tasks.rados.rados.0.smithi159.stdout:652: writing smithi15938319-42 from 608420 to 1137831 tid 1 2024-09-13T13:50:19.863 INFO:tasks.rados.rados.0.smithi159.stdout:652: writing smithi15938319-42 from 1778096 to 2339107 tid 2 2024-09-13T13:50:19.867 INFO:tasks.rados.rados.0.smithi159.stdout:652: writing smithi15938319-42 from 2881850 to 3008420 tid 3 2024-09-13T13:50:19.867 INFO:tasks.rados.rados.0.smithi159.stdout:653: set_chunk oid 47 target oid 2024-09-13T13:50:19.872 INFO:tasks.rados.rados.0.smithi159.stdout:653: set_chunk oid 47 offset: 1121792 length: 658432 target oid 659037d9490718c236d68c25e38d8f0017859e10c8459f84028f62e28387e670 offset: 0 snap 2 2024-09-13T13:50:19.873 INFO:tasks.rados.rados.0.smithi159.stdout:654: read oid 32 snap 1 2024-09-13T13:50:19.873 INFO:tasks.rados.rados.0.smithi159.stdout:654: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-13T13:50:19.873 INFO:tasks.rados.rados.0.smithi159.stdout:655: read oid 2 snap 2 2024-09-13T13:50:19.873 INFO:tasks.rados.rados.0.smithi159.stdout:655: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T13:50:19.873 INFO:tasks.rados.rados.0.smithi159.stdout:656: set_chunk oid 22 target oid 2024-09-13T13:50:19.878 INFO:tasks.rados.rados.0.smithi159.stdout:656: set_chunk oid 22 offset: 386048 length: 596480 target oid f744530bea7030f6f4d0839ab0ce85da4a232544f9267b6d8a95e784b91ad4ae offset: 0 snap -1 2024-09-13T13:50:19.879 INFO:tasks.rados.rados.0.smithi159.stdout:657: delete oid 45 current snap is 3 2024-09-13T13:50:19.879 INFO:tasks.rados.rados.0.smithi159.stdout:645: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:19.879 INFO:tasks.rados.rados.0.smithi159.stdout:645: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:19.879 INFO:tasks.rados.rados.0.smithi159.stdout:645: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:19.879 INFO:tasks.rados.rados.0.smithi159.stdout:646: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:19.932 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:19.932 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:19.933 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:19.932+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:19.941 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:19.964 INFO:tasks.rados.rados.0.smithi159.stdout:645: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:19.964 INFO:tasks.rados.rados.0.smithi159.stdout:647: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:19.970 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:19.971 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:19.986 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 817623 2024-09-13T13:50:19.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:19.993 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:20.001 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.025 INFO:tasks.rados.rados.0.smithi159.stdout:648: finishing set_chunk to oid 49 2024-09-13T13:50:20.025 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 793088 length: 884736 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: finishing write tid 5 to smithi15938319-14 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: oid 14 updating version 0 to 28 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: oid 14 version 28 is already newer than 26 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: oid 14 updating version 28 to 29 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: oid 14 version 29 is already newer than 27 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 29 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:645: left oid 14 (ObjNum 59 snap 3 seq_num 59) 2024-09-13T13:50:20.026 INFO:tasks.rados.rados.0.smithi159.stdout:649: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T13:50:20.033 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:20.033+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:20.084 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.084 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:20.110 INFO:tasks.rados.rados.0.smithi159.stdout:650: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:20.134 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:20.133+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:644: got (0) Success 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:651: finishing set_chunk to oid 28 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 1350656 length: 880640 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:20.176 INFO:tasks.rados.rados.0.smithi159.stdout:652: finishing write tid 1 to smithi15938319-42 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: finishing write tid 2 to smithi15938319-42 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: finishing write tid 3 to smithi15938319-42 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: finishing write tid 4 to smithi15938319-42 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:653: finishing set_chunk to oid 47 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 1121792 length: 658432 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: finishing write tid 5 to smithi15938319-42 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: oid 42 updating version 0 to 22 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: oid 42 version 22 is already newer than 21 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: oid 42 version 22 is already newer than 19 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: oid 42 version 22 is already newer than 20 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 22 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-09-13T13:50:20.177 INFO:tasks.rados.rados.0.smithi159.stdout:652: left oid 42 (ObjNum 60 snap 3 seq_num 60) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:644: done (13 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:645: done (12 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:646: done (11 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:647: done (10 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:648: done (9 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:649: done (8 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:650: done (7 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:651: done (6 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:652: done (5 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:653: done (4 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:657: done (3 left) 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:658: read oid 14 snap -1 2024-09-13T13:50:20.178 INFO:tasks.rados.rados.0.smithi159.stdout:658: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-13T13:50:20.179 INFO:tasks.rados.rados.0.smithi159.stdout:659: read oid 12 snap -1 2024-09-13T13:50:20.179 INFO:tasks.rados.rados.0.smithi159.stdout:659: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:20.179 INFO:tasks.rados.rados.0.smithi159.stdout:660: delete oid 27 current snap is 3 2024-09-13T13:50:20.179 INFO:tasks.rados.rados.0.smithi159.stdout:654: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-13T13:50:20.179 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1466618 2024-09-13T13:50:20.222 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.235 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:20.234+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:20.240 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.240 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:20.250 INFO:tasks.rados.rados.0.smithi159.stdout:656: finishing set_chunk to oid 22 2024-09-13T13:50:20.250 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 9 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T13:50:20.250 INFO:tasks.rados.rados.0.smithi159.stdout:655: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T13:50:20.252 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1259521 2024-09-13T13:50:20.273 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1778175 2024-09-13T13:50:20.290 INFO:tasks.rados.rados.0.smithi159.stdout:659: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-13T13:50:20.291 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.334 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1115693 2024-09-13T13:50:20.336 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:20.335+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:20.345 INFO:tasks.rados.rados.0.smithi159.stdout:658: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-13T13:50:20.345 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.389 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.392 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.392 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.400 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:20.437 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:20.436+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:654: done (5 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:655: done (4 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:656: done (3 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:658: done (2 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:659: done (1 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:660: done (0 left) 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:661: rollback oid 12 current snap is 3 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 12 to 2 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:662: read oid 42 snap 2 2024-09-13T13:50:20.479 INFO:tasks.rados.rados.0.smithi159.stdout:662: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:663: read oid 43 snap -1 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:663: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:664: rollback oid 48 current snap is 3 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 48 to 2 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:665: read oid 10 snap -1 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:665: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:666: read oid 19 snap -1 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:666: expect deleted 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:667: write oid 25 current snap is 3 2024-09-13T13:50:20.480 INFO:tasks.rados.rados.0.smithi159.stdout:667: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-13T13:50:20.484 INFO:tasks.rados.rados.0.smithi159.stdout:667: writing smithi15938319-25 from 658841 to 1130352 tid 1 2024-09-13T13:50:20.490 INFO:tasks.rados.rados.0.smithi159.stdout:667: writing smithi15938319-25 from 1553542 to 2261880 tid 2 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:667: writing smithi15938319-25 from 2964119 to 3058841 tid 3 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:668: read oid 22 snap -1 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:668: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:669: delete oid 27 current snap is 3 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:661: finishing rollback tid 0 to smithi15938319-12 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:661: finishing rollback tid 1 to smithi15938319-12 2024-09-13T13:50:20.495 INFO:tasks.rados.rados.0.smithi159.stdout:661: finishing rollback tid 2 to smithi15938319-12 2024-09-13T13:50:20.496 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 31 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T13:50:20.496 INFO:tasks.rados.rados.0.smithi159.stdout:664: finishing rollback tid 0 to smithi15938319-48 2024-09-13T13:50:20.496 INFO:tasks.rados.rados.0.smithi159.stdout:664: finishing rollback tid 1 to smithi15938319-48 2024-09-13T13:50:20.496 INFO:tasks.rados.rados.0.smithi159.stdout:662: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T13:50:20.513 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1259393 2024-09-13T13:50:20.532 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.538 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:20.537+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:20.547 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.547 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:20.561 INFO:tasks.rados.rados.0.smithi159.stdout:667: finishing write tid 1 to smithi15938319-25 2024-09-13T13:50:20.561 INFO:tasks.rados.rados.0.smithi159.stdout:663: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T13:50:20.618 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2545905 2024-09-13T13:50:20.627 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.640 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:20.639+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:20.675 INFO:tasks.rados.rados.0.smithi159.stdout:665: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:20.695 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.699 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.699 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:20.739 INFO:tasks.rados.rados.0.smithi159.stdout:664: finishing rollback tid 2 to smithi15938319-48 2024-09-13T13:50:20.739 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 38 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T13:50:20.739 INFO:tasks.rados.rados.0.smithi159.stdout:667: finishing write tid 2 to smithi15938319-25 2024-09-13T13:50:20.739 INFO:tasks.rados.rados.0.smithi159.stdout:667: finishing write tid 3 to smithi15938319-25 2024-09-13T13:50:20.739 INFO:tasks.rados.rados.0.smithi159.stdout:667: finishing write tid 4 to smithi15938319-25 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: finishing write tid 5 to smithi15938319-25 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: oid 25 updating version 0 to 29 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: oid 25 version 29 is already newer than 27 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: oid 25 version 29 is already newer than 26 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: oid 25 version 29 is already newer than 28 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 29 (ObjNum 63 snap 3 seq_num 63) dirty exists 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:667: left oid 25 (ObjNum 63 snap 3 seq_num 63) 2024-09-13T13:50:20.740 INFO:tasks.rados.rados.0.smithi159.stdout:668: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:20.741 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:20.738+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:20.820 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 903127 2024-09-13T13:50:20.841 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:20.840+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:20.848 INFO:tasks.rados.rados.0.smithi159.stdout:661: done (8 left) 2024-09-13T13:50:20.848 INFO:tasks.rados.rados.0.smithi159.stdout:662: done (7 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:663: done (6 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:664: done (5 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:665: done (4 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:666: done (3 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:667: done (2 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:668: done (1 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:669: done (0 left) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:670: read oid 50 snap -1 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:670: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:671: read oid 18 snap 2 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:671: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T13:50:20.849 INFO:tasks.rados.rados.0.smithi159.stdout:672: set_chunk oid 9 target oid 2024-09-13T13:50:20.850 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.850 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:20.866 INFO:tasks.rados.rados.0.smithi159.stdout:672: set_chunk oid 9 offset: 458752 length: 1058304 target oid 14cd9c9b8c5237d9a9b98fd416eee3ade31a6a3656ee47ef2c1248e3d806b93e offset: 0 snap -1 2024-09-13T13:50:20.866 INFO:tasks.rados.rados.0.smithi159.stdout:673: read oid 5 snap -1 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:673: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:674: read oid 46 snap 1 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:674: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:675: read oid 24 snap -1 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:675: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:676: tier_promote oid 28 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:670: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:20.867 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:20.932 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2719665 2024-09-13T13:50:20.938 INFO:tasks.rados.rados.0.smithi159.stdout:671: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T13:50:20.942 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1797355 2024-09-13T13:50:20.942 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:20.941+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:20.960 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:20.960 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:20.969 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:20.996 INFO:tasks.rados.rados.0.smithi159.stdout:672: finishing set_chunk to oid 9 2024-09-13T13:50:20.996 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 458752 length: 1058304 2024-09-13T13:50:20.996 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T13:50:20.996 INFO:tasks.rados.rados.0.smithi159.stdout:674: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:20.996 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.043 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:21.042+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:21.072 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.073 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.079 INFO:tasks.rados.rados.0.smithi159.stdout:673: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:21.081 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:21.101 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.125 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.144 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:21.143+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:21.148 INFO:tasks.rados.rados.0.smithi159.stdout:676: got (0) Success 2024-09-13T13:50:21.148 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:21.148 INFO:tasks.rados.rados.0.smithi159.stdout:675: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:21.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:50:21.185 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.186 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.186 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:21.239 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:21.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:21.240 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-13T13:50:21.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-09-13T13:50:21.278 INFO:tasks.rados.rados.0.smithi159.stdout:670: done (6 left) 2024-09-13T13:50:21.278 INFO:tasks.rados.rados.0.smithi159.stdout:671: done (5 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:672: done (4 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:673: done (3 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:674: done (2 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:675: done (1 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:676: done (0 left) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:677: read oid 24 snap 2 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:677: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:678: read oid 27 snap -1 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:678: expect deleted 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:679: tier_promote oid 1 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:680: read oid 22 snap -1 2024-09-13T13:50:21.279 INFO:tasks.rados.rados.0.smithi159.stdout:680: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:21.280 INFO:tasks.rados.rados.0.smithi159.stdout:681: write oid 11 current snap is 3 2024-09-13T13:50:21.280 INFO:tasks.rados.rados.0.smithi159.stdout:681: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-13T13:50:21.284 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:21.283+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:21.285 INFO:tasks.rados.rados.0.smithi159.stdout:681: writing smithi15938319-11 from 692455 to 1314464 tid 1 2024-09-13T13:50:21.293 INFO:tasks.rados.rados.0.smithi159.stdout:681: writing smithi15938319-11 from 2070506 to 2782513 tid 2 2024-09-13T13:50:21.293 INFO:tasks.rados.rados.0.smithi159.stdout:682: rollback oid 29 current snap is 3 2024-09-13T13:50:21.293 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 29 to 1 2024-09-13T13:50:21.293 INFO:tasks.rados.rados.0.smithi159.stdout:680: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T13:50:21.318 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2720752 2024-09-13T13:50:21.319 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.341 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.341 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.347 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:21.346+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:21.350 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:21.395 INFO:tasks.rados.rados.0.smithi159.stdout:679: got (0) Success 2024-09-13T13:50:21.395 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:21.395 INFO:tasks.rados.rados.0.smithi159.stdout:677: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-13T13:50:21.405 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1882164 2024-09-13T13:50:21.432 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:21.448 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:21.447+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:21.460 INFO:tasks.rados.rados.0.smithi159.stdout:681: finishing write tid 1 to smithi15938319-11 2024-09-13T13:50:21.461 INFO:tasks.rados.rados.0.smithi159.stdout:677: done (5 left) 2024-09-13T13:50:21.461 INFO:tasks.rados.rados.0.smithi159.stdout:678: done (4 left) 2024-09-13T13:50:21.461 INFO:tasks.rados.rados.0.smithi159.stdout:679: done (3 left) 2024-09-13T13:50:21.461 INFO:tasks.rados.rados.0.smithi159.stdout:680: done (2 left) 2024-09-13T13:50:21.461 INFO:tasks.rados.rados.0.smithi159.stdout:683: snap_remove snap 2 2024-09-13T13:50:21.494 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.494 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:21.548 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:21.547+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:21.566 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-13T13:50:21.564+0000 7fe36cfd9640 -1 mon.a@0(leader).osd e46 definitely_dead 0 2024-09-13T13:50:21.647 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.647 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.649 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:21.649+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:21.655 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:21.750 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:21.750+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:21.758 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.758 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:21.851 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:21.850+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:21.870 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:21.870 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:21.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:21.963 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:21.962+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:22.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:22.022 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.022 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.032 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:22.053 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:22.052+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:22.145 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:22.154 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:22.152+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:50:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.6. 2024-09-13T13:50:22.156 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:22.155+0000 7f2284f10640 -1 osd.10 46 handle_osd_map injecting map crc failure 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:683: done (2 left) 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:684: read oid 15 snap -1 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:684: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:685: rollback oid 39 current snap is 3 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 39 to 1 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:682: finishing rollback tid 0 to smithi15938319-29 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:681: finishing write tid 2 to smithi15938319-11 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:682: finishing rollback tid 1 to smithi15938319-29 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:682: finishing rollback tid 2 to smithi15938319-29 2024-09-13T13:50:22.160 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 42 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:681: finishing write tid 3 to smithi15938319-11 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:682: done (3 left) 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:686: read oid 10 snap -1 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:686: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:687: read oid 35 snap 1 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:687: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:688: tier_evict oid 45 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:688: tier_evict oid 45 snap -1 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:681: finishing write tid 4 to smithi15938319-11 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:681: oid 11 updating version 0 to 31 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:681: oid 11 updating version 31 to 32 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:681: oid 11 updating version 32 to 33 2024-09-13T13:50:22.161 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 33 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-09-13T13:50:22.162 INFO:tasks.rados.rados.0.smithi159.stdout:681: left oid 11 (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:22.162 INFO:tasks.rados.rados.0.smithi159.stdout:681: done (5 left) 2024-09-13T13:50:22.162 INFO:tasks.rados.rados.0.smithi159.stdout:689: write oid 37 current snap is 3 2024-09-13T13:50:22.162 INFO:tasks.rados.rados.0.smithi159.stdout:689: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-13T13:50:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:689: writing smithi15938319-37 from 709262 to 1406520 tid 1 2024-09-13T13:50:22.175 INFO:tasks.rados.rados.0.smithi159.stdout:689: writing smithi15938319-37 from 2012635 to 2584653 tid 2 2024-09-13T13:50:22.177 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:22.178 INFO:tasks.rados.rados.0.smithi159.stdout:689: writing smithi15938319-37 from 3046388 to 3109262 tid 3 2024-09-13T13:50:22.178 INFO:tasks.rados.rados.0.smithi159.stdout:690: read oid 33 snap -1 2024-09-13T13:50:22.178 INFO:tasks.rados.rados.0.smithi159.stdout:690: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:22.178 INFO:tasks.rados.rados.0.smithi159.stdout:691: read oid 9 snap -1 2024-09-13T13:50:22.179 INFO:tasks.rados.rados.0.smithi159.stdout:691: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:22.179 INFO:tasks.rados.rados.0.smithi159.stdout:692: delete oid 41 current snap is 3 2024-09-13T13:50:22.179 INFO:tasks.rados.rados.0.smithi159.stdout:688: got (22) Invalid argument 2024-09-13T13:50:22.179 INFO:tasks.rados.rados.0.smithi159.stdout:686: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:22.180 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.180 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.181 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1383879 2024-09-13T13:50:22.190 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:22.197 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:22.197 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:22.218 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:22.235 INFO:tasks.rados.rados.0.smithi159.stdout:685: finishing rollback tid 0 to smithi15938319-39 2024-09-13T13:50:22.235 INFO:tasks.rados.rados.0.smithi159.stdout:687: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:22.235 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:22.241 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:22.255 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:22.254+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:22.261 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:22.261 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:22.267 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1398639 2024-09-13T13:50:22.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:22.282 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:22.295 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.295 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.303 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:22.303 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:22.304 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:22.314 INFO:tasks.rados.rados.0.smithi159.stdout:685: finishing rollback tid 1 to smithi15938319-39 2024-09-13T13:50:22.314 INFO:tasks.rados.rados.0.smithi159.stdout:685: finishing rollback tid 2 to smithi15938319-39 2024-09-13T13:50:22.315 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 36 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:22.315 INFO:tasks.rados.rados.0.smithi159.stdout:691: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:22.323 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:22.339 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:22.345 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:22.345 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:22.356 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:22.356+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:22.368 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:22.368 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:22.390 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:22.391 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:22.409 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.409 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.410 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:22.411 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:22.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:22.457 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:22.457+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:22.559 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:22.558+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:22.561 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.561 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.569 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:22.635 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:22.634+0000 7efcfb7fe640 0 -- 172.21.15.27:0/3671576328 >> v2:172.21.15.122:6808/1255553857 conn(0x7efcfc221ee0 msgr2=0x7efcfc2242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:22.660 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:22.659+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:22.672 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:22.672 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:22.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:22.761 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:22.760+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:22.789 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:22.789 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:22.789 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:22.789 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:22.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:22.859 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:22.860 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:22.861 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:22.860+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:22.881 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:22.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:22.910 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:22.910 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:22.910 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:22.910 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:22.910 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:22.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:22.962 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:22.961+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:23.063 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:23.062+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4290.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-13T13:50:23.067 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:49:50.409235+0000", 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.657815405000001, 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069561700000000005, 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409235+0000", 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409235+0000", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409235+0000", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8e-07 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409243+0000", 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.216e-06 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409269+0000", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6290000000000001e-05 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409279+0000", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6690000000000001e-06 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409329+0000", 2024-09-13T13:50:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0126000000000001e-05 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409353+0000", 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3782e-05 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409902+0000", 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054904599999999997 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:49:50.409931+0000", 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9308000000000001e-05 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.072 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:23.161 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:23.161+0000 7f3c37938640 -1 osd.5 47 handle_osd_map injecting map crc failure 2024-09-13T13:50:23.164 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:23.163+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:23.229 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.229 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:23.229 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:23.229 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:23.248 INFO:tasks.rados.rados.0.smithi159.stdout:689: finishing write tid 1 to smithi15938319-37 2024-09-13T13:50:23.252 INFO:tasks.rados.rados.0.smithi159.stdout:689: finishing write tid 2 to smithi15938319-37 2024-09-13T13:50:23.252 INFO:tasks.rados.rados.0.smithi159.stdout:689: finishing write tid 3 to smithi15938319-37 2024-09-13T13:50:23.252 INFO:tasks.rados.rados.0.smithi159.stdout:689: finishing write tid 4 to smithi15938319-37 2024-09-13T13:50:23.252 INFO:tasks.rados.rados.0.smithi159.stdout:689: finishing write tid 5 to smithi15938319-37 2024-09-13T13:50:23.252 INFO:tasks.rados.rados.0.smithi159.stdout:689: oid 37 updating version 0 to 34 2024-09-13T13:50:23.253 INFO:tasks.rados.rados.0.smithi159.stdout:689: oid 37 updating version 34 to 35 2024-09-13T13:50:23.253 INFO:tasks.rados.rados.0.smithi159.stdout:689: oid 37 updating version 35 to 36 2024-09-13T13:50:23.253 INFO:tasks.rados.rados.0.smithi159.stdout:689: oid 37 updating version 36 to 37 2024-09-13T13:50:23.253 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 37 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-09-13T13:50:23.253 INFO:tasks.rados.rados.0.smithi159.stdout:689: left oid 37 (ObjNum 66 snap 3 seq_num 66) 2024-09-13T13:50:23.266 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:23.265+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:23.346 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.347 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:23.347 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:23.347 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:23.347 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:23.399 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:23.398+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:23.449 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:23.448+0000 7fe857f4f640 -1 osd.7 47 handle_osd_map injecting map crc failure 2024-09-13T13:50:23.470 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:23.468+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:199 4.e 4:709ad093:::smithi15938319-50:head [setxattr _header (56) in=7b,truncate 2823543] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.732792+0000", 2024-09-13T13:50:23.473 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.731992658999999, 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007448143, 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 199 2024-09-13T13:50:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732792+0000", 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732792+0000", 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732792+0000", 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.27e-07 2024-09-13T13:50:23.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732795+0000", 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6589999999999999e-06 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732800+0000", 2024-09-13T13:50:23.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.972e-06 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732804+0000", 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0709999999999996e-06 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.733660+0000", 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000856359 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.733722+0000", 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1760000000000005e-05 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.733830+0000", 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000108027 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734379+0000", 2024-09-13T13:50:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054878999999999995 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740122+0000", 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005743382 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740183+0000", 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1148000000000005e-05 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740240+0000", 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6548000000000001e-05 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:200 4.e 4:709ad093:::smithi15938319-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.731959003, 2024-09-13T13:50:23.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075516929999999999, 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 200 2024-09-13T13:50:23.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732826+0000", 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2300000000000002e-07 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732828+0000", 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.328e-06 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732833+0000", 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7269999999999998e-06 2024-09-13T13:50:23.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732838+0000", 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2460000000000003e-06 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734037+0000", 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00119889 2024-09-13T13:50:23.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734078+0000", 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1675000000000003e-05 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740269+0000", 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00619018 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740318+0000", 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9521999999999998e-05 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740377+0000", 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8801999999999997e-05 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.490 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:224 3.1 e41/33 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.780033+0000", 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.684750847, 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067280630000000003, 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.780033+0000", 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.780033+0000", 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.780034+0000", 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5900000000000001e-07 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783659+0000", 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003625465 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783688+0000", 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9451e-05 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783696+0000", 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3780000000000003e-06 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783750+0000", 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4156000000000001e-05 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783781+0000", 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1170000000000001e-05 2024-09-13T13:50:23.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.786707+0000", 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002925567 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.786761+0000", 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4716999999999998e-05 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:217 3.1 e41/33 3:90dfe591:::smithi15938319-18 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.680792113999999, 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097209170000000008, 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783993+0000", 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:50:23.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787823+0000", 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038302620000000001 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787880+0000", 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6681999999999997e-05 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787888+0000", 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6270000000000006e-06 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787940+0000", 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2132000000000002e-05 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787966+0000", 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5741999999999998e-05 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793657+0000", 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056903739999999998 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793713+0000", 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6449000000000001e-05 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:223 3.1 e41/33 3:932c05fb:::smithi15938319-24 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.676953879999999, 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009035576, 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787831+0000", 2024-09-13T13:50:23.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790933+0000", 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031022459999999999 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791063+0000", 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129374 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791071+0000", 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0020000000000006e-06 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791120+0000", 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9348999999999998e-05 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791135+0000", 2024-09-13T13:50:23.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5108e-05 2024-09-13T13:50:23.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796837+0000", 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057013999999999997 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796866+0000", 2024-09-13T13:50:23.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9295999999999999e-05 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:225 3.1 e41/33 3:b64df3d0:::smithi15938319-26 low_tier:head v 41'4, mlcod=0'0)", 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.790939+0000", 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.673845543000001, 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0070734999999999999, 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.507 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790939+0000", 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790939+0000", 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790940+0000", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3099999999999996e-07 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793682+0000", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002742389 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793712+0000", 2024-09-13T13:50:23.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9692999999999998e-05 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793719+0000", 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.951e-06 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793759+0000", 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9950999999999997e-05 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793775+0000", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6008e-05 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797988+0000", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042131319999999996 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.798012+0000", 2024-09-13T13:50:23.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4445000000000001e-05 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:228 3.1 e41/33 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'5, mlcod=0'0)", 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.667005742000001, 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011341255999999999, 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797780+0000", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.283e-06 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800442+0000", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026622740000000001 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800472+0000", 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9765999999999999e-05 2024-09-13T13:50:23.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800477+0000", 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2429999999999996e-06 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800515+0000", 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7750000000000003e-05 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800533+0000", 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8657000000000001e-05 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809070+0000", 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085367659999999995 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809120+0000", 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9517e-05 2024-09-13T13:50:23.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:220 3.1 e41/33 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.660638795000001, 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015696446999999999, 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804147+0000", 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1629999999999999e-06 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810603+0000", 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064567410000000002 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810700+0000", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6526000000000004e-05 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810708+0000", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9009999999999994e-06 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810749+0000", 2024-09-13T13:50:23.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810767+0000", 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7798000000000001e-05 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819801+0000", 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090344989999999997 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819842+0000", 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.053e-05 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:231 3.1 e41/33 3:9f3f9b00:::smithi15938319-32 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.654172495999999, 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012065741, 2024-09-13T13:50:23.522 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810613+0000", 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8400000000000002e-07 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813879+0000", 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003266658 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813938+0000", 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8579000000000003e-05 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813945+0000", 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3540000000000002e-06 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813991+0000", 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5203000000000001e-05 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.814007+0000", 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6850999999999998e-05 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822644+0000", 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086365629999999999 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822677+0000", 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3549e-05 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:232 3.1 e41/33 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'8, mlcod=41'4)", 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.650899854, 2024-09-13T13:50:23.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088135090000000006, 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813885+0000", 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6899999999999997e-07 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817413+0000", 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035275020000000001 2024-09-13T13:50:23.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817568+0000", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155131 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817574+0000", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4479999999999999e-06 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817639+0000", 2024-09-13T13:50:23.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5221000000000005e-05 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817657+0000", 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7450000000000001e-05 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822683+0000", 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050262070000000004 2024-09-13T13:50:23.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822698+0000", 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4881e-05 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:235 3.1 e41/33 3:9761ca94:::smithi15938319-36 low_tier:head v 41'9, mlcod=41'4)", 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.647364826, 2024-09-13T13:50:23.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090169459999999996, 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:23.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817420+0000", 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2300000000000001e-07 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821383+0000", 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039627890000000004 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821417+0000", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4496999999999999e-05 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821424+0000", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0319999999999998e-06 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821470+0000", 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5748000000000003e-05 2024-09-13T13:50:23.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821488+0000", 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8207e-05 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826398+0000", 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004909409 2024-09-13T13:50:23.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826436+0000", 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8640999999999998e-05 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.537 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:242 3.1 e41/33 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'8)", 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.633300030999999, 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083574370000000005, 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831485+0000", 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2099999999999995e-07 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834193+0000", 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027077939999999999 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834220+0000", 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7018000000000001e-05 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834229+0000", 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7849999999999997e-06 2024-09-13T13:50:23.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834275+0000", 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5961000000000003e-05 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834296+0000", 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1401999999999999e-05 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839806+0000", 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055102500000000004 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839842+0000", 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5305999999999997e-05 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:258 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.566600893, 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011133136, 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898185+0000", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7400000000000001e-06 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898191+0000", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898205+0000", 2024-09-13T13:50:23.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4628e-05 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898213+0000", 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5020000000000003e-06 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898410+0000", 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000197235 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898490+0000", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9856000000000003e-05 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.904838+0000", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063482139999999996 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.905236+0000", 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039744999999999998 2024-09-13T13:50:23.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909250+0000", 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040139110000000002 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909281+0000", 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078999999999998e-05 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909316+0000", 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5778999999999997e-05 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:248 3.1 e41/33 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'13, mlcod=41'11)", 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.557586097, 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012034909999999999, 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907199+0000", 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4499999999999996e-07 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911325+0000", 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041262089999999996 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911350+0000", 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4960000000000002e-05 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911356+0000", 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4770000000000002e-06 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911406+0000", 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0633000000000001e-05 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911422+0000", 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5937999999999999e-05 2024-09-13T13:50:23.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919153+0000", 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077307970000000002 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919233+0000", 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0050999999999996e-05 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.554 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:267 4.e 4:721f6cf4:::smithi15938319-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.549905598, 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075444659999999997, 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.555 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 267 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914880+0000", 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914888+0000", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0039999999999999e-06 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914901+0000", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2738999999999999e-05 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914910+0000", 2024-09-13T13:50:23.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4700000000000008e-06 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914965+0000", 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4695999999999999e-05 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.915028+0000", 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2921000000000003e-05 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921586+0000", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065584730000000004 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922280+0000", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069410100000000003 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922365+0000", 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4581999999999998e-05 2024-09-13T13:50:23.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922394+0000", 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8997999999999999e-05 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922423+0000", 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9162e-05 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:299 4.e 4:709ad093:::smithi15938319-50:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.6953455030000004, 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089831609999999999, 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 299 2024-09-13T13:50:23.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769441+0000", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5830000000000001e-06 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769453+0000", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1756000000000001e-05 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769498+0000", 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4468999999999997e-05 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769510+0000", 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7052000000000005e-05 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769700+0000", 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2683999999999996e-05 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777412+0000", 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077117100000000001 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777907+0000", 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049540900000000004 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778307+0000", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039943500000000001 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778372+0000", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5153000000000005e-05 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778422+0000", 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9941000000000002e-05 2024-09-13T13:50:23.567 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:358 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.3903211930000001, 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086044410000000009, 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.568 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 358 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:23.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074464+0000", 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074474+0000", 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5670000000000001e-06 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074488+0000", 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3889e-05 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074497+0000", 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074595+0000", 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7899999999999994e-05 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074675+0000", 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9849000000000001e-05 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077944+0000", 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003269044 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082062+0000", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041176290000000003 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082985+0000", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092279199999999997 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083021+0000", 2024-09-13T13:50:23.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6386e-05 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083067+0000", 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6483999999999997e-05 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:367 4.e 4:721f6cf4:::smithi15938319-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:50:23.574 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.3898509289999996, 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0071188880000000003, 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 367 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074934+0000", 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.034e-06 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074940+0000", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0279999999999997e-06 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074955+0000", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4918e-05 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074967+0000", 2024-09-13T13:50:23.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1438e-05 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.075027+0000", 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0421000000000003e-05 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.075109+0000", 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2090999999999997e-05 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077119+0000", 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020096100000000002 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080158+0000", 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0030388939999999999 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.081944+0000", 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00178673 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.081999+0000", 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5009000000000001e-05 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082052+0000", 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2714999999999999e-05 2024-09-13T13:50:23.580 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:807 4.e 4:717ade62:::smithi15938319-24:head [write 541192~628415,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.283967123, 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010708927, 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.581 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 807 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180820+0000", 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7879999999999998e-06 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181481+0000", 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066089999999999996 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181501+0000", 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0503e-05 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181511+0000", 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9329999999999996e-06 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181582+0000", 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0916999999999999e-05 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181588+0000", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1160000000000004e-06 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185876+0000", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042876900000000003 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185941+0000", 2024-09-13T13:50:23.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5168999999999999e-05 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.186064+0000", 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012253599999999999 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191138+0000", 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050743660000000003 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191450+0000", 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031216700000000002 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191481+0000", 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0954000000000002e-05 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191526+0000", 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4888000000000001e-05 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:864 4.e 4:70cd1372:::smithi15938319-9:head [read 569344~1232896] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 4.3187268769999996, 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075254359999999999, 2024-09-13T13:50:23.588 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 864 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146060+0000", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9720000000000001e-06 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146071+0000", 2024-09-13T13:50:23.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0806e-05 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146094+0000", 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2883000000000001e-05 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146105+0000", 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0668e-05 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146170+0000", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5060000000000004e-05 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146174+0000", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5990000000000004e-06 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.152136+0000", 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0059615049999999998 2024-09-13T13:50:23.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.152240+0000", 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103903 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.153583+0000", 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00134304 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.594 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.595 INFO:tasks.rados.rados.0.smithi159.stdout:690: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:23.595 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:23.595 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:23.595 INFO:tasks.rados.rados.0.smithi159.stdout:684: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:23.596 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:23.594+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:23.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:23.673 INFO:tasks.rados.rados.0.smithi159.stdout:684: done (8 left) 2024-09-13T13:50:23.673 INFO:tasks.rados.rados.0.smithi159.stdout:685: done (7 left) 2024-09-13T13:50:23.673 INFO:tasks.rados.rados.0.smithi159.stdout:686: done (6 left) 2024-09-13T13:50:23.673 INFO:tasks.rados.rados.0.smithi159.stdout:687: done (5 left) 2024-09-13T13:50:23.673 INFO:tasks.rados.rados.0.smithi159.stdout:688: done (4 left) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:689: done (3 left) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:690: done (2 left) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:691: done (1 left) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:692: done (0 left) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:693: read oid 39 snap -1 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:693: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:694: write oid 14 current snap is 3 2024-09-13T13:50:23.674 INFO:tasks.rados.rados.0.smithi159.stdout:694: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-13T13:50:23.679 INFO:tasks.rados.rados.0.smithi159.stdout:694: writing smithi15938319-14 from 742876 to 1190632 tid 1 2024-09-13T13:50:23.685 INFO:tasks.rados.rados.0.smithi159.stdout:694: writing smithi15938319-14 from 1729599 to 2305286 tid 2 2024-09-13T13:50:23.689 INFO:tasks.rados.rados.0.smithi159.stdout:694: writing smithi15938319-14 from 3101234 to 3142876 tid 3 2024-09-13T13:50:23.689 INFO:tasks.rados.rados.0.smithi159.stdout:695: write (excl) oid 7 current snap is 3 2024-09-13T13:50:23.689 INFO:tasks.rados.rados.0.smithi159.stdout:695: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-13T13:50:23.695 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:23.695+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:23.696 INFO:tasks.rados.rados.0.smithi159.stdout:695: writing smithi15938319-7 from 759683 to 1399041 tid 1 2024-09-13T13:50:23.703 INFO:tasks.rados.rados.0.smithi159.stdout:695: writing smithi15938319-7 from 1904434 to 2623779 tid 2 2024-09-13T13:50:23.706 INFO:tasks.rados.rados.0.smithi159.stdout:695: writing smithi15938319-7 from 3128657 to 3159683 tid 3 2024-09-13T13:50:23.706 INFO:tasks.rados.rados.0.smithi159.stdout:696: read oid 5 snap 1 2024-09-13T13:50:23.706 INFO:tasks.rados.rados.0.smithi159.stdout:696: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:23.706 INFO:tasks.rados.rados.0.smithi159.stdout:697: tier_promote oid 41 2024-09-13T13:50:23.706 INFO:tasks.rados.rados.0.smithi159.stdout:693: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.748 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:23.748 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:23.748 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:23.761 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2395753 2024-09-13T13:50:23.764 INFO:tasks.rados.rados.0.smithi159.stdout:694: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: finishing write tid 5 to smithi15938319-14 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: oid 14 updating version 0 to 42 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: oid 14 updating version 42 to 43 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: oid 14 version 43 is already newer than 40 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: oid 14 version 43 is already newer than 41 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 43 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: left oid 14 (ObjNum 68 snap 3 seq_num 68) 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:695: finishing write tid 1 to smithi15938319-7 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:693: done (4 left) 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:694: done (3 left) 2024-09-13T13:50:23.765 INFO:tasks.rados.rados.0.smithi159.stdout:698: write (excl) oid 32 current snap is 3 2024-09-13T13:50:23.766 INFO:tasks.rados.rados.0.smithi159.stdout:695: finishing write tid 2 to smithi15938319-7 2024-09-13T13:50:23.766 INFO:tasks.rados.rados.0.smithi159.stdout:695: finishing write tid 3 to smithi15938319-7 2024-09-13T13:50:23.766 INFO:tasks.rados.rados.0.smithi159.stdout:698: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-13T13:50:23.772 INFO:tasks.rados.rados.0.smithi159.stdout:698: writing smithi15938319-32 from 776490 to 1491097 tid 1 2024-09-13T13:50:23.780 INFO:tasks.rados.rados.0.smithi159.stdout:698: writing smithi15938319-32 from 2246563 to 2825919 tid 2 2024-09-13T13:50:23.780 INFO:tasks.rados.rados.0.smithi159.stdout:699: read oid 11 snap -1 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:699: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:700: read oid 9 snap 1 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:700: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:701: read oid 37 snap -1 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:701: expect (ObjNum 66 snap 3 seq_num 66) 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:702: tier_promote oid 10 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:697: got (0) Success 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 33 (ObjNum 273 snap 0 seq_num 0) clean dne 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:695: finishing write tid 4 to smithi15938319-7 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:695: finishing write tid 5 to smithi15938319-7 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:695: oid 7 updating version 0 to 41 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:695: oid 7 version 41 is already newer than 40 2024-09-13T13:50:23.781 INFO:tasks.rados.rados.0.smithi159.stdout:695: oid 7 updating version 41 to 43 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:695: oid 7 version 43 is already newer than 42 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 43 (ObjNum 69 snap 3 seq_num 69) dirty exists 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:695: left oid 7 (ObjNum 69 snap 3 seq_num 69) 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:695: done (7 left) 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:697: done (6 left) 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:703: read oid 47 snap -1 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:703: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:704: tier_evict oid 4 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:704: tier_evict oid 4 snap -1 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:696: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:23.782 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:23.796 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:23.796+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:23.804 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:23.837 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1847994 2024-09-13T13:50:23.838 INFO:tasks.rados.rados.0.smithi159.stdout:698: finishing write tid 1 to smithi15938319-32 2024-09-13T13:50:23.838 INFO:tasks.rados.rados.0.smithi159.stdout:704: got (22) Invalid argument 2024-09-13T13:50:23.838 INFO:tasks.rados.rados.0.smithi159.stdout:700: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:23.866 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:23.866 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:23.866 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:23.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:23.894 INFO:tasks.rados.rados.0.smithi159.stdout:698: finishing write tid 2 to smithi15938319-32 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: finishing write tid 3 to smithi15938319-32 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: finishing write tid 4 to smithi15938319-32 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: oid 32 updating version 0 to 15 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: oid 32 updating version 15 to 16 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: oid 32 version 16 is already newer than 14 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 16 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: left oid 32 (ObjNum 70 snap 3 seq_num 70) 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:696: done (7 left) 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:698: done (6 left) 2024-09-13T13:50:23.895 INFO:tasks.rados.rados.0.smithi159.stdout:700: done (5 left) 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:704: done (4 left) 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:705: read oid 1 snap -1 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:705: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:706: delete oid 22 current snap is 3 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:702: got (0) Success 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:699: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:23.896 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 534927 2024-09-13T13:50:23.897 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:23.896+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.818673969000001, 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:50:23.986 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:50:23.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:23.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.818647975999999, 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:23.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:23.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:50:23.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.998 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:19 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.813702455, 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096180398, 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:23.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170583+0000", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170585+0000", 2024-09-13T13:50:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1090000000000001e-06 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266590+0000", 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096005323000000004 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266598+0000", 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8110000000000003e-06 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266650+0000", 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2481000000000001e-05 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266658+0000", 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266763+0000", 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105209 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.713339154, 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:50:24.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:50:24.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:111 4.2 e41/34 4:4d382039:::smithi15938319-28:head v 41'9, mlcod=41'5)", 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.540299255000001, 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.06900017, 2024-09-13T13:50:24.008 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1600000000000001e-07 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443991+0000", 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512563+0000", 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.068572151999999997 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512567+0000", 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7119999999999999e-06 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512607+0000", 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249000000000003e-05 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512623+0000", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5866999999999999e-05 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512961+0000", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033803199999999999 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512986+0000", 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5079000000000002e-05 2024-09-13T13:50:24.012 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:135 4.a 4:5004af91:::smithi15938319-34:head [setxattr _header (56) in=7b,truncate 2554631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:24.013 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.470378414000001, 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034151982999999997, 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 135 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513907+0000", 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.99e-07 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513910+0000", 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2210000000000002e-06 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513922+0000", 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1858999999999999e-05 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513929+0000", 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1069999999999999e-06 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514737+0000", 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080765899999999996 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514809+0000", 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1928999999999995e-05 2024-09-13T13:50:24.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514899+0000", 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0511000000000001e-05 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515444+0000", 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054460299999999995 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547991+0000", 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032546974999999999 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548022+0000", 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1442999999999997e-05 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548058+0000", 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6077e-05 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.019 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:136 4.a 4:5004af91:::smithi15938319-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.470337419, 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034245790999999998, 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.020 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 136 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513948+0000", 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0200000000000002e-07 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513951+0000", 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2909999999999999e-06 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513958+0000", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7950000000000002e-06 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513964+0000", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.874e-06 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515064+0000", 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001099865 2024-09-13T13:50:24.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515106+0000", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2398000000000003e-05 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548087+0000", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032980559 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548133+0000", 2024-09-13T13:50:24.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6343e-05 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548193+0000", 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0064e-05 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.025 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:135 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.468596152, 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032374238999999999, 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8099999999999999e-07 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515690+0000", 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7000000000000003e-07 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547939+0000", 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032248700999999998 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547943+0000", 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6990000000000002e-06 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547981+0000", 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7302999999999998e-05 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547985+0000", 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000002e-06 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548063+0000", 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7894999999999999e-05 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:149 4.a 4:5c84836d:::smithi15938319-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.426104128, 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011167603, 2024-09-13T13:50:24.030 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 149 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558181+0000", 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558182+0000", 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1799999999999999e-06 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558629+0000", 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044714800000000002 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558640+0000", 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0631999999999999e-05 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558645+0000", 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0379999999999999e-06 2024-09-13T13:50:24.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558697+0000", 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1931000000000001e-05 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558781+0000", 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4166999999999996e-05 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.558864+0000", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2762999999999993e-05 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.559564+0000", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00070006799999999998 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569279+0000", 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097149720000000005 2024-09-13T13:50:24.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569319+0000", 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0296000000000002e-05 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569348+0000", 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9408e-05 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.036 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.417303977, 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:24.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:50:24.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:50:24.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.417289477000001, 2024-09-13T13:50:24.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:24.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.415036037, 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.049 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:50:24.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.053 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.292612749, 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:24.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:50:24.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:50:24.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.292581930000001, 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.060 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:50:24.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.290423821999999, 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:24.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:209 3.3 e41/33 3:c3255dc7:::smithi15938319-10 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.226970884, 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012424223, 2024-09-13T13:50:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757314+0000", 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.757315+0000", 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6799999999999999e-07 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759545+0000", 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002230793 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759568+0000", 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2237000000000001e-05 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759574+0000", 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.618e-06 2024-09-13T13:50:24.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759615+0000", 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1078999999999998e-05 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.759631+0000", 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5914e-05 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.074 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.769699+0000", 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010068027 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.769738+0000", 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8787e-05 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:236 3.3 e41/33 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'10, mlcod=41'6)", 2024-09-13T13:50:24.075 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.177213183999999, 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012626577999999999, 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.807072+0000", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5999999999999999e-07 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810513+0000", 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034412800000000001 2024-09-13T13:50:24.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810543+0000", 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0054999999999999e-05 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810552+0000", 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810615+0000", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2788999999999998e-05 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810643+0000", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7535999999999999e-05 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819621+0000", 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089780380000000007 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819698+0000", 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7553999999999996e-05 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.080889623999999, 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:50:24.080 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.084 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:50:24.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.214912092, 2024-09-13T13:50:24.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:24.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:50:24.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.201236378000001, 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:50:24.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:50:24.098 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.099 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:24.101 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:24.100+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:24.101 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2656632 2024-09-13T13:50:24.101 INFO:tasks.rados.rados.0.smithi159.stdout:701: expect (ObjNum 66 snap 3 seq_num 66) 2024-09-13T13:50:24.101 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 623381 2024-09-13T13:50:24.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:24.108 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.155 INFO:tasks.rados.rados.0.smithi159.stdout:703: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:24.159 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1497887 2024-09-13T13:50:24.177 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 735752 2024-09-13T13:50:24.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:50:24.201 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:24.201+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:24.210 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:24.210 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:24.210 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:24.210 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:24.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:24.232 INFO:tasks.rados.rados.0.smithi159.stdout:705: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T13:50:24.232 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1819761 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:699: done (5 left) 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:701: done (4 left) 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:702: done (3 left) 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:703: done (2 left) 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:705: done (1 left) 2024-09-13T13:50:24.289 INFO:tasks.rados.rados.0.smithi159.stdout:706: done (0 left) 2024-09-13T13:50:24.290 INFO:tasks.rados.rados.0.smithi159.stdout:707: write oid 26 current snap is 3 2024-09-13T13:50:24.290 INFO:tasks.rados.rados.0.smithi159.stdout:707: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-13T13:50:24.292 INFO:tasks.rados.rados.0.smithi159.stdout:707: writing smithi15938319-26 from 410104 to 875209 tid 1 2024-09-13T13:50:24.299 INFO:tasks.rados.rados.0.smithi159.stdout:707: writing smithi15938319-26 from 1563527 to 2146552 tid 2 2024-09-13T13:50:24.303 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:24.302+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:24.305 INFO:tasks.rados.rados.0.smithi159.stdout:707: writing smithi15938319-26 from 2810926 to 3210104 tid 3 2024-09-13T13:50:24.305 INFO:tasks.rados.rados.0.smithi159.stdout:708: tier_evict oid 46 2024-09-13T13:50:24.305 INFO:tasks.rados.rados.0.smithi159.stdout:708: tier_evict oid 46 snap -1 2024-09-13T13:50:24.305 INFO:tasks.rados.rados.0.smithi159.stdout:709: write oid 25 current snap is 3 2024-09-13T13:50:24.305 INFO:tasks.rados.rados.0.smithi159.stdout:709: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-13T13:50:24.311 INFO:tasks.rados.rados.0.smithi159.stdout:709: writing smithi15938319-25 from 426911 to 967265 tid 1 2024-09-13T13:50:24.318 INFO:tasks.rados.rados.0.smithi159.stdout:709: writing smithi15938319-25 from 1622009 to 2348692 tid 2 2024-09-13T13:50:24.323 INFO:tasks.rados.rados.0.smithi159.stdout:709: writing smithi15938319-25 from 3121996 to 3226911 tid 3 2024-09-13T13:50:24.323 INFO:tasks.rados.rados.0.smithi159.stdout:710: read oid 29 snap -1 2024-09-13T13:50:24.323 INFO:tasks.rados.rados.0.smithi159.stdout:710: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T13:50:24.323 INFO:tasks.rados.rados.0.smithi159.stdout:711: set_chunk oid 16 target oid 2024-09-13T13:50:24.334 INFO:tasks.rados.rados.0.smithi159.stdout:711: set_chunk oid 16 offset: 1043968 length: 934400 target oid c06b0b7466f37065fc151ae6dbba0970ed23ed87f8d94dc3bd6953e733bb6b20 offset: 0 snap -1 2024-09-13T13:50:24.334 INFO:tasks.rados.rados.0.smithi159.stdout:712: rollback oid 14 current snap is 3 2024-09-13T13:50:24.334 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 14 to 1 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:707: finishing write tid 1 to smithi15938319-26 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:707: finishing write tid 2 to smithi15938319-26 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:713: read oid 19 snap -1 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:713: expect deleted 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:714: write (excl) oid 23 current snap is 3 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:708: got (22) Invalid argument 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:707: finishing write tid 3 to smithi15938319-26 2024-09-13T13:50:24.335 INFO:tasks.rados.rados.0.smithi159.stdout:714: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-13T13:50:24.339 INFO:tasks.rados.rados.0.smithi159.stdout:714: writing smithi15938319-23 from 443718 to 1059321 tid 1 2024-09-13T13:50:24.344 INFO:tasks.rados.rados.0.smithi159.stdout:714: writing smithi15938319-23 from 1564138 to 2034479 tid 2 2024-09-13T13:50:24.351 INFO:tasks.rados.rados.0.smithi159.stdout:714: writing smithi15938319-23 from 2633066 to 3243718 tid 3 2024-09-13T13:50:24.351 INFO:tasks.rados.rados.0.smithi159.stdout:708: done (7 left) 2024-09-13T13:50:24.351 INFO:tasks.rados.rados.0.smithi159.stdout:715: read oid 28 snap -1 2024-09-13T13:50:24.351 INFO:tasks.rados.rados.0.smithi159.stdout:715: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T13:50:24.351 INFO:tasks.rados.rados.0.smithi159.stdout:716: set_chunk oid 39 target oid 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:716: set_chunk oid 39 offset: 315392 length: 59904 target oid 1293a1139466aae4fdfd990d192ecae0dce690a6ce116429611c1ffefa14ee06 offset: 0 snap -1 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:717: read oid 9 snap 1 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:717: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:707: finishing write tid 4 to smithi15938319-26 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:709: finishing write tid 1 to smithi15938319-25 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:709: finishing write tid 2 to smithi15938319-25 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:718: read oid 49 snap -1 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:718: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:719: write oid 12 current snap is 3 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:709: finishing write tid 3 to smithi15938319-25 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:709: finishing write tid 4 to smithi15938319-25 2024-09-13T13:50:24.354 INFO:tasks.rados.rados.0.smithi159.stdout:709: finishing write tid 5 to smithi15938319-25 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:709: oid 25 updating version 0 to 39 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:709: oid 25 version 39 is already newer than 37 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:709: oid 25 version 39 is already newer than 36 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:709: oid 25 version 39 is already newer than 38 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 39 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:709: left oid 25 (ObjNum 73 snap 3 seq_num 73) 2024-09-13T13:50:24.355 INFO:tasks.rados.rados.0.smithi159.stdout:719: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-13T13:50:24.359 INFO:tasks.rados.rados.0.smithi159.stdout:719: writing smithi15938319-12 from 460525 to 1151377 tid 1 2024-09-13T13:50:24.366 INFO:tasks.rados.rados.0.smithi159.stdout:719: writing smithi15938319-12 from 1622620 to 2352972 tid 2 2024-09-13T13:50:24.371 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:24.371 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:24.371 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:24.371 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:24.371 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:24.372 INFO:tasks.rados.rados.0.smithi159.stdout:719: writing smithi15938319-12 from 3060489 to 3260525 tid 3 2024-09-13T13:50:24.372 INFO:tasks.rados.rados.0.smithi159.stdout:709: done (11 left) 2024-09-13T13:50:24.372 INFO:tasks.rados.rados.0.smithi159.stdout:720: read oid 4 snap -1 2024-09-13T13:50:24.372 INFO:tasks.rados.rados.0.smithi159.stdout:720: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T13:50:24.372 INFO:tasks.rados.rados.0.smithi159.stdout:721: delete oid 30 current snap is 3 2024-09-13T13:50:24.373 INFO:tasks.rados.rados.0.smithi159.stdout:710: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-13T13:50:24.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:24.422 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:24.421+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:24.441 INFO:tasks.rados.rados.0.smithi159.stdout:711: finishing set_chunk to oid 16 2024-09-13T13:50:24.441 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 1043968 length: 934400 2024-09-13T13:50:24.441 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:712: finishing rollback tid 0 to smithi15938319-14 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:712: finishing rollback tid 1 to smithi15938319-14 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:710: done (12 left) 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:711: done (11 left) 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:713: done (10 left) 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:721: done (9 left) 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:722: write (excl) oid 20 current snap is 3 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:712: finishing rollback tid 2 to smithi15938319-14 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T13:50:24.442 INFO:tasks.rados.rados.0.smithi159.stdout:714: finishing write tid 1 to smithi15938319-23 2024-09-13T13:50:24.443 INFO:tasks.rados.rados.0.smithi159.stdout:714: finishing write tid 2 to smithi15938319-23 2024-09-13T13:50:24.443 INFO:tasks.rados.rados.0.smithi159.stdout:722: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-13T13:50:24.447 INFO:tasks.rados.rados.0.smithi159.stdout:722: writing smithi15938319-20 from 494139 to 1051842 tid 1 2024-09-13T13:50:24.453 INFO:tasks.rados.rados.0.smithi159.stdout:722: writing smithi15938319-20 from 1455937 to 2189958 tid 2 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:722: writing smithi15938319-20 from 2948041 to 3294139 tid 3 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:712: done (9 left) 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:723: read oid 11 snap -1 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:723: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:724: read oid 17 snap -1 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:724: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:725: rollback oid 31 current snap is 3 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 31 to 1 2024-09-13T13:50:24.460 INFO:tasks.rados.rados.0.smithi159.stdout:716: finishing set_chunk to oid 39 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 315392 length: 59904 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: finishing write tid 3 to smithi15938319-23 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: finishing write tid 4 to smithi15938319-23 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: finishing write tid 5 to smithi15938319-23 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: oid 23 updating version 0 to 30 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: oid 23 version 30 is already newer than 29 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: oid 23 updating version 30 to 31 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: oid 23 updating version 31 to 32 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 32 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: left oid 23 (ObjNum 74 snap 3 seq_num 74) 2024-09-13T13:50:24.461 INFO:tasks.rados.rados.0.smithi159.stdout:714: done (11 left) 2024-09-13T13:50:24.462 INFO:tasks.rados.rados.0.smithi159.stdout:716: done (10 left) 2024-09-13T13:50:24.462 INFO:tasks.rados.rados.0.smithi159.stdout:726: delete oid 37 current snap is 3 2024-09-13T13:50:24.462 INFO:tasks.rados.rados.0.smithi159.stdout:717: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:24.462 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.504 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:24.503+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:21 4.d 4:b771c1ad:::smithi15938319-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.353485916, 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010356456999999999, 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175385+0000", 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.238e-06 2024-09-13T13:50:24.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176269+0000", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088463099999999998 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176280+0000", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0831e-05 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176289+0000", 2024-09-13T13:50:24.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176354+0000", 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5102000000000001e-05 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176360+0000", 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0929999999999999e-06 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180157+0000", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003796982 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180282+0000", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012503899999999999 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180445+0000", 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016250300000000001 2024-09-13T13:50:24.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184519+0000", 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040741090000000002 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185660+0000", 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001141186 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185696+0000", 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6143000000000003e-05 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185740+0000", 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3634000000000003e-05 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:22 4.d 4:b771c1ad:::smithi15938319-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.538 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.35256502, 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097627319999999997, 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 22 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176304+0000", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176305+0000", 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7700000000000004e-07 2024-09-13T13:50:24.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176334+0000", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8750000000000001e-05 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176341+0000", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1350000000000003e-06 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176347+0000", 2024-09-13T13:50:24.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6110000000000001e-06 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176428+0000", 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1902000000000003e-05 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176431+0000", 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8590000000000001e-06 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181669+0000", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052375959999999997 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181723+0000", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4209999999999998e-05 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.181816+0000", 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2446000000000002e-05 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184783+0000", 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029669269999999998 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185959+0000", 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011760799999999999 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186016+0000", 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7726999999999998e-05 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186067+0000", 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0812000000000001e-05 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:23 4.d 4:b771c1ad:::smithi15938319-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.545 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.352511821, 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0098262599999999999, 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 23 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.546 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176358+0000", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176362+0000", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4750000000000002e-06 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176368+0000", 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:50:24.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176373+0000", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1279999999999999e-06 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176463+0000", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9957999999999995e-05 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176469+0000", 2024-09-13T13:50:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.925e-06 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182113+0000", 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056435290000000004 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182187+0000", 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3993999999999997e-05 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182323+0000", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136122 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184790+0000", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024670999999999998 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186114+0000", 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001323765 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186142+0000", 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8067000000000001e-05 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186184+0000", 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1922000000000003e-05 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.552 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:24 4.d 4:b771c1ad:::smithi15938319-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.352485911, 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010036734, 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9100000000000004e-07 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176387+0000", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6060000000000001e-06 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176409+0000", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2161e-05 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176417+0000", 2024-09-13T13:50:24.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2309999999999996e-06 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176484+0000", 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7453999999999995e-05 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176486+0000", 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.95e-06 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182553+0000", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060667480000000003 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182596+0000", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2447000000000002e-05 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186229+0000", 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036334179999999998 2024-09-13T13:50:24.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186308+0000", 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8979000000000004e-05 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186420+0000", 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112249 2024-09-13T13:50:24.559 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:46 4.c 4:34c00adb:::smithi15938319-12:head [write 1610596~562003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.283726088, 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0051361310000000004, 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.560 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 46 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245143+0000", 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245144+0000", 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2600000000000001e-07 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245643+0000", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049847100000000001 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245654+0000", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1473e-05 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245660+0000", 2024-09-13T13:50:24.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5450000000000003e-06 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245716+0000", 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5908000000000001e-05 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245803+0000", 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7570000000000004e-05 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.245941+0000", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013756599999999999 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.247677+0000", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0017366440000000001 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250179+0000", 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025012039999999999 2024-09-13T13:50:24.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250235+0000", 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6073000000000003e-05 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.250279+0000", 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4750999999999999e-05 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:174 4.c 4:34327e96:::smithi15938319-44:head [write 1816608~788001] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.875516814000001, 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.005761937, 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 174 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.567 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653353+0000", 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.653354+0000", 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0359999999999999e-06 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654276+0000", 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092214800000000002 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654286+0000", 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0251000000000001e-05 2024-09-13T13:50:24.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654298+0000", 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1576999999999999e-05 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654371+0000", 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3044000000000001e-05 2024-09-13T13:50:24.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654420+0000", 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9453000000000002e-05 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.654488+0000", 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7547999999999998e-05 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.655731+0000", 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012433909999999999 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659062+0000", 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033305679999999999 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659091+0000", 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9448999999999999e-05 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.659115+0000", 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3472000000000002e-05 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.3.22:13 4.c 4:358f784f:::smithi15938319-41:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.71110951, 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092837100000000006, 2024-09-13T13:50:24.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817760+0000", 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817761+0000", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4529999999999999e-06 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817768+0000", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.764e-06 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817788+0000", 2024-09-13T13:50:24.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9417000000000001e-05 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817795+0000", 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4619999999999999e-06 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822308+0000", 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045129460000000003 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822421+0000", 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00011317800000000001 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.827044+0000", 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046224899999999999 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.577 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.630715778000001, 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:50:24.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:50:24.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.630647921, 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:50:24.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.617364995999999, 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.591 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:50:24.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:311 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.759192334, 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0091613149999999997, 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.597 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 311 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769677+0000", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769678+0000", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.19e-06 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769687+0000", 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5210000000000002e-06 2024-09-13T13:50:24.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769699+0000", 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2346e-05 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769708+0000", 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769790+0000", 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2278000000000001e-05 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769891+0000", 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101059 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777673+0000", 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077819710000000004 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778488+0000", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00081508000000000004 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778779+0000", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00029088900000000001 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778807+0000", 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8079e-05 2024-09-13T13:50:24.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778838+0000", 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1411000000000001e-05 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.454088327999999, 2024-09-13T13:50:24.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:50:24.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:390 4.c 4:358f784f:::smithi15938319-41:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.449262118, 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065600609999999998, 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 390 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079607+0000", 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079609+0000", 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.438e-06 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079614+0000", 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1000000000000003e-06 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079626+0000", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2361000000000001e-05 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079633+0000", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079679+0000", 2024-09-13T13:50:24.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6201e-05 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.079730+0000", 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1313999999999997e-05 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085125+0000", 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053946200000000001 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085782+0000", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00065719300000000003 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086055+0000", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00027300700000000002 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086118+0000", 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2491999999999997e-05 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.086167+0000", 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9811999999999997e-05 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:786 4.c 4:34327e96:::smithi15938319-44:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e44)", 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 7.3946844460000003, 2024-09-13T13:50:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.006240097, 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 786 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134185+0000", 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134187+0000", 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.193e-06 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134195+0000", 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2660000000000001e-06 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134213+0000", 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7999000000000002e-05 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134223+0000", 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2129999999999997e-06 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134283+0000", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0652999999999997e-05 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.134289+0000", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4330000000000003e-06 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.136814+0000", 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002525158 2024-09-13T13:50:24.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.136917+0000", 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000102877 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.137114+0000", 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019731499999999999 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.139063+0000", 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019493749999999999 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140320+0000", 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012565199999999999 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140374+0000", 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4017999999999997e-05 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:17.140425+0000", 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1076999999999999e-05 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:844 4.c 4:34327e96:::smithi15938319-44:head [write 574806~778913] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.650553+0000", 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8783162799999999, 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089605120000000003, 2024-09-13T13:50:24.623 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 844 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.650553+0000", 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.650553+0000", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.650555+0000", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.199e-06 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652102+0000", 2024-09-13T13:50:24.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015470550000000001 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652118+0000", 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5307000000000001e-05 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652127+0000", 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2590000000000006e-06 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652197+0000", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0110000000000005e-05 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652295+0000", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8267e-05 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.652468+0000", 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017227799999999999 2024-09-13T13:50:24.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.654837+0000", 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023691580000000001 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659409+0000", 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045720429999999996 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659468+0000", 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8805999999999999e-05 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659514+0000", 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6029999999999998e-05 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:845 4.c 4:34327e96:::smithi15938319-44:head [write 2061132~557342] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.657346+0000", 2024-09-13T13:50:24.629 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8715229960000004, 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056864899999999998, 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 845 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657346+0000", 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657346+0000", 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657347+0000", 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4900000000000004e-07 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657823+0000", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000475964 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657835+0000", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1428e-05 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657840+0000", 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2120000000000002e-06 2024-09-13T13:50:24.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657897+0000", 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6805000000000002e-05 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657988+0000", 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1036e-05 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.658142+0000", 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015452700000000001 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.662462+0000", 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043191719999999996 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.662935+0000", 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047300199999999998 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.662989+0000", 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4608999999999998e-05 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663033+0000", 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3785999999999997e-05 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:846 4.c 4:34327e96:::smithi15938319-44:head [setxattr _header (56) in=7b,truncate 2974806] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:24.635 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.657853+0000", 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.87101658, 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053342950000000002, 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 846 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657853+0000", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657853+0000", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657853+0000", 2024-09-13T13:50:24.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8500000000000001e-07 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657857+0000", 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2270000000000002e-06 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657863+0000", 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8290000000000002e-06 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657868+0000", 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.301e-06 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659057+0000", 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011888210000000001 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659108+0000", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1487999999999999e-05 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659189+0000", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0783000000000002e-05 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663046+0000", 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038574630000000002 2024-09-13T13:50:24.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663100+0000", 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3770000000000002e-05 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663134+0000", 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3763000000000002e-05 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663187+0000", 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3264999999999999e-05 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:847 4.c 4:34327e96:::smithi15938319-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.657874+0000", 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8709956739999996, 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055384120000000004, 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 847 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657874+0000", 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657874+0000", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657874+0000", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.58e-07 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657889+0000", 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4615e-05 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657896+0000", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7299999999999999e-06 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657904+0000", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.85e-06 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659550+0000", 2024-09-13T13:50:24.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016457690000000001 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.659590+0000", 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9773e-05 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663237+0000", 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003647113 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663314+0000", 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6742000000000005e-05 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663412+0000", 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8362000000000005e-05 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:862 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1110208] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:24.649 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.657946+0000", 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8709233779999996, 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056752419999999996, 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 862 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657946+0000", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657946+0000", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657946+0000", 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4900000000000002e-07 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657947+0000", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1629999999999999e-06 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657950+0000", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8650000000000001e-06 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.657953+0000", 2024-09-13T13:50:24.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6139999999999999e-06 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663442+0000", 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054891560000000002 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663494+0000", 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1464000000000001e-05 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.663621+0000", 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012773099999999999 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1087 4.c 4:34c00adb:::smithi15938319-12:head [write 460525~690852] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 0.16859012600000001, 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010491474000000001, 2024-09-13T13:50:24.654 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1087 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360281+0000", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0999999999999998e-06 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364170+0000", 2024-09-13T13:50:24.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038886929999999999 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364189+0000", 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9185999999999999e-05 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364198+0000", 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364278+0000", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9839999999999995e-05 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364399+0000", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364616+0000", 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021726100000000001 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.366707+0000", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002091341 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370646+0000", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039392430000000003 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370714+0000", 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8069000000000002e-05 2024-09-13T13:50:24.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370771+0000", 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6292000000000002e-05 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:24.660 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:24.662 INFO:tasks.rados.rados.0.smithi159.stdout:715: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T13:50:24.662 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1335989 2024-09-13T13:50:24.662 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.662 INFO:tasks.rados.rados.0.smithi159.stdout:718: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:24.663 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.663 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1572116 2024-09-13T13:50:24.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:24.664 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:24.661+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:24.677 INFO:tasks.rados.rados.0.smithi159.stdout:719: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: finishing write tid 5 to smithi15938319-12 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: oid 12 updating version 0 to 38 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: oid 12 updating version 38 to 40 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: oid 12 version 40 is already newer than 37 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: oid 12 version 40 is already newer than 39 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 40 (ObjNum 75 snap 3 seq_num 75) dirty exists 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:719: left oid 12 (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:720: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T13:50:24.678 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.700 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.733 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1986588 2024-09-13T13:50:24.734 INFO:tasks.rados.rados.0.smithi159.stdout:707: finishing write tid 5 to smithi15938319-26 2024-09-13T13:50:24.734 INFO:tasks.rados.rados.0.smithi159.stdout:707: oid 26 updating version 0 to 22 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:707: oid 26 updating version 22 to 23 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:707: oid 26 version 23 is already newer than 21 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:707: oid 26 version 23 is already newer than 20 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 23 (ObjNum 72 snap 3 seq_num 72) dirty exists 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:707: left oid 26 (ObjNum 72 snap 3 seq_num 72) 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:722: finishing write tid 1 to smithi15938319-20 2024-09-13T13:50:24.735 INFO:tasks.rados.rados.0.smithi159.stdout:722: finishing write tid 2 to smithi15938319-20 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:722: finishing write tid 3 to smithi15938319-20 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:722: finishing write tid 4 to smithi15938319-20 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:707: done (10 left) 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:715: done (9 left) 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:717: done (8 left) 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:718: done (7 left) 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:719: done (6 left) 2024-09-13T13:50:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:720: done (5 left) 2024-09-13T13:50:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:726: done (4 left) 2024-09-13T13:50:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:727: write (excl) oid 19 current snap is 3 2024-09-13T13:50:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:723: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:24.755 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2577468 2024-09-13T13:50:24.762 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:24.762+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:24.777 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:24.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:24.777 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:24.778 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:24.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:24.813 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.859 INFO:tasks.rados.rados.0.smithi159.stdout:725: finishing rollback tid 0 to smithi15938319-31 2024-09-13T13:50:24.859 INFO:tasks.rados.rados.0.smithi159.stdout:725: finishing rollback tid 1 to smithi15938319-31 2024-09-13T13:50:24.859 INFO:tasks.rados.rados.0.smithi159.stdout:724: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:24.859 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:24.863 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:24.863+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:24.887 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1741277 2024-09-13T13:50:24.892 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:24.892 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:24.892 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:24.892 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:24.892 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:24.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:24.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:722: finishing write tid 5 to smithi15938319-20 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:722: oid 20 updating version 0 to 27 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:722: oid 20 updating version 27 to 28 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:722: oid 20 version 28 is already newer than 26 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:722: oid 20 version 28 is already newer than 25 2024-09-13T13:50:24.923 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 28 (ObjNum 77 snap 3 seq_num 77) dirty exists 2024-09-13T13:50:24.924 INFO:tasks.rados.rados.0.smithi159.stdout:722: left oid 20 (ObjNum 77 snap 3 seq_num 77) 2024-09-13T13:50:24.924 INFO:tasks.rados.rados.0.smithi159.stdout:727: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-13T13:50:24.930 INFO:tasks.rados.rados.0.smithi159.stdout:727: writing smithi15938319-19 from 527753 to 1235954 tid 1 2024-09-13T13:50:24.937 INFO:tasks.rados.rados.0.smithi159.stdout:727: writing smithi15938319-19 from 1856548 to 2594238 tid 2 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:727: writing smithi15938319-19 from 3286534 to 3327753 tid 3 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:722: done (4 left) 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:723: done (3 left) 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:724: done (2 left) 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:728: tier_promote oid 24 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:725: finishing rollback tid 2 to smithi15938319-31 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 44 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:727: finishing write tid 1 to smithi15938319-19 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:727: finishing write tid 2 to smithi15938319-19 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:725: done (2 left) 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:729: read oid 46 snap 1 2024-09-13T13:50:24.942 INFO:tasks.rados.rados.0.smithi159.stdout:729: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:24.943 INFO:tasks.rados.rados.0.smithi159.stdout:730: read oid 42 snap -1 2024-09-13T13:50:24.943 INFO:tasks.rados.rados.0.smithi159.stdout:730: expect (ObjNum 60 snap 3 seq_num 60) 2024-09-13T13:50:24.943 INFO:tasks.rados.rados.0.smithi159.stdout:731: set_chunk oid 4 target oid 2024-09-13T13:50:24.952 INFO:tasks.rados.rados.0.smithi159.stdout:731: set_chunk oid 4 offset: 639488 length: 1267200 target oid acdbf3073af3cd64b945e7f3edd32c967b36acb523d4a7b88e28cb0814e3c374 offset: 0 snap -1 2024-09-13T13:50:24.953 INFO:tasks.rados.rados.0.smithi159.stdout:732: set_chunk oid 15 target oid 2024-09-13T13:50:24.964 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:24.963+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:24.966 INFO:tasks.rados.rados.0.smithi159.stdout:732: set_chunk oid 15 offset: 340992 length: 1658880 target oid 30df22e1a48467c834b42e4f60c60067cd7ef4a6bf938e44a68534083ca1a134 offset: 0 snap -1 2024-09-13T13:50:24.966 INFO:tasks.rados.rados.0.smithi159.stdout:733: read oid 9 snap -1 2024-09-13T13:50:24.966 INFO:tasks.rados.rados.0.smithi159.stdout:733: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:24.966 INFO:tasks.rados.rados.0.smithi159.stdout:734: write oid 7 current snap is 3 2024-09-13T13:50:24.967 INFO:tasks.rados.rados.0.smithi159.stdout:734: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-13T13:50:24.973 INFO:tasks.rados.rados.0.smithi159.stdout:734: writing smithi15938319-7 from 544560 to 1328010 tid 1 2024-09-13T13:50:24.979 INFO:tasks.rados.rados.0.smithi159.stdout:734: writing smithi15938319-7 from 1915030 to 2396378 tid 2 2024-09-13T13:50:24.984 INFO:tasks.rados.rados.0.smithi159.stdout:734: writing smithi15938319-7 from 2797604 to 3277365 tid 3 2024-09-13T13:50:24.984 INFO:tasks.rados.rados.0.smithi159.stdout:735: read oid 27 snap -1 2024-09-13T13:50:24.984 INFO:tasks.rados.rados.0.smithi159.stdout:735: expect deleted 2024-09-13T13:50:24.984 INFO:tasks.rados.rados.0.smithi159.stdout:736: rollback oid 32 current snap is 3 2024-09-13T13:50:24.984 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 32 to 1 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:727: finishing write tid 3 to smithi15938319-19 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:737: read oid 47 snap -1 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:737: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:738: read oid 50 snap -1 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:738: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:739: tier_promote oid 33 2024-09-13T13:50:24.985 INFO:tasks.rados.rados.0.smithi159.stdout:729: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:25.035 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:25.040 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 85639 2024-09-13T13:50:25.049 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:25.049 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1 4.7 4:e63fe140:::smithi15938319-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.929708967, 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044289289999999999, 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117794+0000", 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.117795+0000", 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8100000000000001e-07 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118374+0000", 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00057889599999999997 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118388+0000", 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3969e-05 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118399+0000", 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1168999999999999e-05 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118711+0000", 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00031199899999999998 2024-09-13T13:50:25.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118800+0000", 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8660999999999997e-05 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.118899+0000", 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9193999999999999e-05 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120073+0000", 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001174476 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122149+0000", 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020752969999999998 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122184+0000", 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5830000000000001e-05 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.122223+0000", 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8457000000000001e-05 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3 4.7 4:e63fe140:::smithi15938319-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.927363073, 2024-09-13T13:50:25.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045172090000000003, 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120140+0000", 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120141+0000", 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0699999999999996e-07 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120144+0000", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.331e-06 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120150+0000", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1109999999999996e-06 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120155+0000", 2024-09-13T13:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9370000000000003e-06 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121069+0000", 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00091422599999999997 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121156+0000", 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6473999999999999e-05 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121286+0000", 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000130662 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121691+0000", 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00040453800000000002 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124593+0000", 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029016990000000002 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124624+0000", 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1055e-05 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.124657+0000", 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3368999999999998e-05 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4 4.7 4:e63fe140:::smithi15938319-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.927339148, 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058803479999999997, 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.063 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4499999999999997e-07 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120167+0000", 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8700000000000001e-06 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120173+0000", 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2269999999999998e-06 2024-09-13T13:50:25.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120182+0000", 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4819999999999994e-06 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121521+0000", 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001339668 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:25.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121596+0000", 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4987000000000006e-05 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125923+0000", 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043266620000000002 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125983+0000", 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300000000000002e-05 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.126044+0000", 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0707e-05 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:73 4.7 4:e1bd516b:::smithi15938319-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.721772014999999, 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060681499999999996, 2024-09-13T13:50:25.068 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 73 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325734+0000", 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326420+0000", 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068555600000000003 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326435+0000", 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.575e-05 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326444+0000", 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8410000000000006e-06 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326517+0000", 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2784e-05 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326630+0000", 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112759 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326729+0000", 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9761999999999998e-05 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.328491+0000", 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00176133 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331735+0000", 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003244317 2024-09-13T13:50:25.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331768+0000", 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2750000000000003e-05 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331799+0000", 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1281000000000001e-05 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:74 4.7 4:e1bd516b:::smithi15938319-19:head [write 1903617~398909] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.717475104, 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041941720000000004, 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.075 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 74 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330028+0000", 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330029+0000", 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.217e-06 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330323+0000", 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029364499999999998 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330334+0000", 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0903999999999999e-05 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330340+0000", 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0990000000000004e-06 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330413+0000", 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3460999999999993e-05 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330483+0000", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9626e-05 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330556+0000", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3460000000000005e-05 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331402+0000", 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00084570499999999998 2024-09-13T13:50:25.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334176+0000", 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027738569999999998 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334202+0000", 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5732999999999999e-05 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334222+0000", 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0465000000000001e-05 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:75 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.717150062, 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040696860000000003, 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.081 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 75 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330353+0000", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330374+0000", 2024-09-13T13:50:25.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0729000000000001e-05 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330389+0000", 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4990999999999999e-05 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330398+0000", 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.397e-06 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330944+0000", 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054573899999999995 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331015+0000", 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0405000000000001e-05 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331096+0000", 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1872000000000005e-05 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331592+0000", 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00049587600000000002 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334365+0000", 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027724540000000002 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334396+0000", 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1449000000000003e-05 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334423+0000", 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6302e-05 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:76 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.717083592, 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041363490000000001, 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 76 2024-09-13T13:50:25.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330419+0000", 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330420+0000", 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330422+0000", 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5210000000000001e-06 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330429+0000", 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3090000000000001e-06 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.330434+0000", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0300000000000001e-06 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331244+0000", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081012399999999996 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331316+0000", 2024-09-13T13:50:25.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2653999999999999e-05 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334451+0000", 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031348529999999999 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334493+0000", 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1644000000000003e-05 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.334556+0000", 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2714999999999998e-05 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:98 4.7 4:e26a40dd:::smithi15938319-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.642989353000001, 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052413879999999996, 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 98 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2999999999999999e-07 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405146+0000", 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063201000000000002 2024-09-13T13:50:25.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405157+0000", 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0773e-05 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405162+0000", 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2719999999999997e-06 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405203+0000", 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0639000000000002e-05 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405254+0000", 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0760000000000002e-05 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405333+0000", 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9501999999999999e-05 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406334+0000", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010006590000000001 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409690+0000", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033561060000000002 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409720+0000", 2024-09-13T13:50:25.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9771999999999999e-05 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409755+0000", 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5064999999999997e-05 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:99 4.7 4:e26a40dd:::smithi15938319-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:25.099 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.642328527, 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046796340000000002, 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 99 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405174+0000", 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405175+0000", 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405179+0000", 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.653e-06 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405184+0000", 2024-09-13T13:50:25.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.575e-06 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405188+0000", 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7500000000000001e-06 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405851+0000", 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066292300000000005 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405905+0000", 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4012999999999999e-05 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405988+0000", 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3009000000000004e-05 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406517+0000", 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052932299999999995 2024-09-13T13:50:25.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409789+0000", 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032720230000000002 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409821+0000", 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1550000000000001e-05 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409854+0000", 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3327999999999997e-05 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:100 4.7 4:e26a40dd:::smithi15938319-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.642307580000001, 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047806309999999996, 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.108 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 100 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405195+0000", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405196+0000", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1100000000000001e-07 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405198+0000", 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9709999999999998e-06 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405203+0000", 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6629999999999998e-06 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405208+0000", 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6149999999999999e-06 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406140+0000", 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093174499999999999 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406184+0000", 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4008999999999998e-05 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409882+0000", 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036979159999999999 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409921+0000", 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9019e-05 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409976+0000", 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5281999999999997e-05 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.132212315, 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.114 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:50:25.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:50:25.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:50:25.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.127412786000001, 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.120 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:50:25.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.278064748, 2024-09-13T13:50:25.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:50:25.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.973264436999999, 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:50:25.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:50:25.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:50:25.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.8843831929999997, 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:50:25.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:50:25.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:25.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:50:25.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1053 4.7 4:e26a40dd:::smithi15938319-25:head [write 426911~540354] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.311032+0000", 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.736470509, 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035045039999999999, 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.146 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1053 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311032+0000", 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311032+0000", 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311035+0000", 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8629999999999999e-06 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311664+0000", 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062919199999999999 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311682+0000", 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7084000000000001e-05 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311691+0000", 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8779999999999993e-06 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311804+0000", 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011239399999999999 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.311889+0000", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5457999999999994e-05 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.312017+0000", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012777099999999999 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.313831+0000", 2024-09-13T13:50:25.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018142760000000001 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.314458+0000", 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062639100000000001 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.314497+0000", 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9233999999999997e-05 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.314537+0000", 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9962999999999997e-05 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1054 4.7 4:e26a40dd:::smithi15938319-25:head [write 1622009~726683] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.318475+0000", 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.72902750699999996, 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003229418, 2024-09-13T13:50:25.152 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1054 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.318475+0000", 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.318475+0000", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.318477+0000", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0690000000000001e-06 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319108+0000", 2024-09-13T13:50:25.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063104100000000005 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319122+0000", 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4139e-05 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319128+0000", 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319185+0000", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6910000000000002e-05 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319251+0000", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5637999999999997e-05 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.319373+0000", 2024-09-13T13:50:25.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012223600000000001 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.320952+0000", 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015793890000000001 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.321623+0000", 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00067098699999999995 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.321657+0000", 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3658000000000002e-05 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.321705+0000", 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7876e-05 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1111 4.7 4:e1bd516b:::smithi15938319-19:head [write 527753~708201,stat] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.11724287, 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060230190000000001, 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1111 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.159 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930262+0000", 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1229999999999998e-06 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930922+0000", 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065976499999999998 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930939+0000", 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7158000000000001e-05 2024-09-13T13:50:25.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930948+0000", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931023+0000", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5562000000000006e-05 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931094+0000", 2024-09-13T13:50:25.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0445999999999996e-05 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931198+0000", 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010367899999999999 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.932418+0000", 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012203419999999999 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936222+0000", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038036860000000001 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936256+0000", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4477000000000002e-05 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936283+0000", 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6970000000000001e-05 2024-09-13T13:50:25.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1114 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3327753] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.941510+0000", 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.10599314999999999, 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0037720359999999999, 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:25.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1114 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941510+0000", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941510+0000", 2024-09-13T13:50:25.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941510+0000", 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941515+0000", 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.493e-06 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941521+0000", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941529+0000", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9480000000000008e-06 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942046+0000", 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00051721999999999998 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942092+0000", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5145999999999998e-05 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942167+0000", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5708000000000001e-05 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.943167+0000", 2024-09-13T13:50:25.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099940500000000008 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945209+0000", 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020422610000000001 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945242+0000", 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2790999999999998e-05 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945282+0000", 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0058999999999997e-05 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1115 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.941539+0000", 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.10596401900000001, 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038895819999999999, 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1115 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941539+0000", 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941539+0000", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941539+0000", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941543+0000", 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7890000000000001e-06 2024-09-13T13:50:25.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941549+0000", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941554+0000", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2240000000000003e-06 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942326+0000", 2024-09-13T13:50:25.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077158499999999998 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942368+0000", 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2361000000000003e-05 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945313+0000", 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029444689999999999 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945359+0000", 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6391999999999999e-05 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.945429+0000", 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9215999999999996e-05 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:25.176 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:25.178 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:728: got (0) Success 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 27 (ObjNum 56 snap 3 seq_num 56) dirty exists 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: finishing write tid 4 to smithi15938319-19 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: finishing write tid 5 to smithi15938319-19 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: oid 19 updating version 0 to 40 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: oid 19 updating version 40 to 42 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: oid 19 updating version 42 to 43 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: oid 19 version 43 is already newer than 41 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 43 (ObjNum 79 snap 3 seq_num 79) dirty exists 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:727: left oid 19 (ObjNum 79 snap 3 seq_num 79) 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:730: expect (ObjNum 60 snap 3 seq_num 60) 2024-09-13T13:50:25.179 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:25.180 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:25.181 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:25.181 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:25.177+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:25.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:25.183 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:25.184 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:50:25.184 INFO:tasks.rados.rados.0.smithi159.stdout:731: finishing set_chunk to oid 4 2024-09-13T13:50:25.184 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 639488 length: 1267200 2024-09-13T13:50:25.185 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T13:50:25.185 INFO:tasks.rados.rados.0.smithi159.stdout:732: finishing set_chunk to oid 15 2024-09-13T13:50:25.185 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 340992 length: 1658880 2024-09-13T13:50:25.185 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T13:50:25.185 INFO:tasks.rados.rados.0.smithi159.stdout:733: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:25.186 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:25.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:25.207 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:25.208 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:25.228 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:25.228 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:25.243 INFO:tasks.rados.rados.0.smithi159.stdout:734: finishing write tid 1 to smithi15938319-7 2024-09-13T13:50:25.243 INFO:tasks.rados.rados.0.smithi159.stdout:734: finishing write tid 2 to smithi15938319-7 2024-09-13T13:50:25.243 INFO:tasks.rados.rados.0.smithi159.stdout:734: finishing write tid 3 to smithi15938319-7 2024-09-13T13:50:25.243 INFO:tasks.rados.rados.0.smithi159.stdout:734: finishing write tid 4 to smithi15938319-7 2024-09-13T13:50:25.244 INFO:tasks.rados.rados.0.smithi159.stdout:736: finishing rollback tid 0 to smithi15938319-32 2024-09-13T13:50:25.244 INFO:tasks.rados.rados.0.smithi159.stdout:734: finishing write tid 5 to smithi15938319-7 2024-09-13T13:50:25.244 INFO:tasks.rados.rados.0.smithi159.stdout:734: oid 7 updating version 0 to 47 2024-09-13T13:50:25.244 INFO:tasks.rados.rados.0.smithi159.stdout:734: oid 7 updating version 47 to 48 2024-09-13T13:50:25.244 INFO:tasks.rados.rados.0.smithi159.stdout:734: oid 7 updating version 48 to 49 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:734: oid 7 version 49 is already newer than 46 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 49 (ObjNum 80 snap 3 seq_num 80) dirty exists 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:734: left oid 7 (ObjNum 80 snap 3 seq_num 80) 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:736: finishing rollback tid 1 to smithi15938319-32 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:727: done (12 left) 2024-09-13T13:50:25.245 INFO:tasks.rados.rados.0.smithi159.stdout:728: done (11 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:729: done (10 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:730: done (9 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:731: done (8 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:732: done (7 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:733: done (6 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:734: done (5 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:735: done (4 left) 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:740: snap_create 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:736: finishing rollback tid 2 to smithi15938319-32 2024-09-13T13:50:25.246 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 19 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:25.247 INFO:tasks.rados.rados.0.smithi159.stdout:738: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:25.249 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:25.250 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:25.267 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2741886 2024-09-13T13:50:25.270 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:25.278 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:25.278+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:25.291 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:25.291 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:25.303 INFO:tasks.rados.rados.0.smithi159.stdout:737: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:25.303 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:25.312 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:25.317 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:25.316+0000 7f7c71c1d640 -1 osd.14 48 handle_osd_map injecting map crc failure 2024-09-13T13:50:25.325 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.325 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:25.379 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:25.378+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:25.394 INFO:tasks.rados.rados.0.smithi159.stdout:739: got (0) Success 2024-09-13T13:50:25.394 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T13:50:25.477 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.477 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.480 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:25.479+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:25.485 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:25.581 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:25.580+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:25.587 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.587 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:25.671 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:25.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:25.681 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:25.680+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:25.693 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:25.693 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:25.698 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.698 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:25.710 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:25.708+0000 7f1f35760640 -1 osd.4 48 handle_osd_map injecting map crc failure 2024-09-13T13:50:25.782 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:25.782+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:25.809 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.809 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.818 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:25.883 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:25.882+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:25.962 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:25.962 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:25.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:25.985 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:25.984+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:26.073 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.074 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:26.085 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:26.084+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:736: done (4 left) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:737: done (3 left) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:738: done (2 left) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:739: done (1 left) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:740: done (0 left) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:741: read oid 15 snap -1 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:741: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:742: read oid 21 snap -1 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:742: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T13:50:26.178 INFO:tasks.rados.rados.0.smithi159.stdout:743: read oid 13 snap 3 2024-09-13T13:50:26.179 INFO:tasks.rados.rados.0.smithi159.stdout:743: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:26.179 INFO:tasks.rados.rados.0.smithi159.stdout:744: snap_remove snap 1 2024-09-13T13:50:26.186 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:26.184+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:26.188 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.188 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:26.287 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:26.286+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:26.299 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.299 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:26.388 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:26.387+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:26.427 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.428 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:26.489 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:26.488+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:26.540 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.541 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.549 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:26.591 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:26.590+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:26.691 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:26.690+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:26.693 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.693 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:26.792 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:26.791+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:26.805 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.805 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:26.894 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:26.893+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:26.917 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:26.917 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:26.925 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:26.995 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:26.994+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:27.028 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.028 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.037 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:27.096 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:27.095+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:27.177 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [6] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:27.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:27.178 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-13T13:50:27.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-09-13T13:50:27.180 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.181 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.181 INFO:tasks.rados.rados.0.smithi159.stdout:744: done (3 left) 2024-09-13T13:50:27.182 INFO:tasks.rados.rados.0.smithi159.stdout:745: write (excl) oid 18 current snap is 4 2024-09-13T13:50:27.182 INFO:tasks.rados.rados.0.smithi159.stdout:743: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:27.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:50:27.190 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:27.197 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:27.196+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:27.254 INFO:tasks.rados.rados.0.smithi159.stdout:741: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:27.254 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:27.296 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.296 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.297 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:27.296+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:27.305 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:27.321 INFO:tasks.rados.rados.0.smithi159.stdout:742: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T13:50:27.337 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:27.369 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 947731 2024-09-13T13:50:27.384 INFO:tasks.rados.rados.0.smithi159.stdout:745: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-13T13:50:27.390 INFO:tasks.rados.rados.0.smithi159.stdout:745: writing smithi15938319-18 from 561367 to 1020066 tid 1 2024-09-13T13:50:27.396 INFO:tasks.rados.rados.0.smithi159.stdout:745: writing smithi15938319-18 from 1573512 to 2314871 tid 2 2024-09-13T13:50:27.397 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:27.397+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:745: writing smithi15938319-18 from 2941380 to 3361367 tid 3 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:741: done (3 left) 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:742: done (2 left) 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:743: done (1 left) 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:746: read oid 16 snap 3 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:746: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:747: read oid 11 snap -1 2024-09-13T13:50:27.403 INFO:tasks.rados.rados.0.smithi159.stdout:747: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:748: read oid 40 snap -1 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:748: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:749: tier_promote oid 5 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:745: finishing write tid 1 to smithi15938319-18 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:750: read oid 12 snap -1 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:750: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:751: write (excl) oid 14 current snap is 4 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:745: finishing write tid 2 to smithi15938319-18 2024-09-13T13:50:27.404 INFO:tasks.rados.rados.0.smithi159.stdout:751: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-13T13:50:27.408 INFO:tasks.rados.rados.0.smithi159.stdout:751: writing smithi15938319-14 from 578174 to 1112122 tid 1 2024-09-13T13:50:27.413 INFO:tasks.rados.rados.0.smithi159.stdout:751: writing smithi15938319-14 from 1515641 to 2000658 tid 2 2024-09-13T13:50:27.421 INFO:tasks.rados.rados.0.smithi159.stdout:751: writing smithi15938319-14 from 2736097 to 3238402 tid 3 2024-09-13T13:50:27.421 INFO:tasks.rados.rados.0.smithi159.stdout:752: write (excl) oid 31 current snap is 4 2024-09-13T13:50:27.421 INFO:tasks.rados.rados.0.smithi159.stdout:752: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-13T13:50:27.427 INFO:tasks.rados.rados.0.smithi159.stdout:752: writing smithi15938319-31 from 594981 to 1204178 tid 1 2024-09-13T13:50:27.435 INFO:tasks.rados.rados.0.smithi159.stdout:752: writing smithi15938319-31 from 1974123 to 2719151 tid 2 2024-09-13T13:50:27.438 INFO:tasks.rados.rados.0.smithi159.stdout:752: writing smithi15938319-31 from 3279873 to 3394981 tid 3 2024-09-13T13:50:27.438 INFO:tasks.rados.rados.0.smithi159.stdout:753: delete oid 4 current snap is 4 2024-09-13T13:50:27.438 INFO:tasks.rados.rados.0.smithi159.stdout:747: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:27.453 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.454 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.460 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2746372 2024-09-13T13:50:27.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:27.492 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2423474 2024-09-13T13:50:27.499 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:27.497+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:27.519 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-13T13:50:27.518+0000 7fe36cfd9640 -1 mon.a@0(leader).osd e51 definitely_dead 0 2024-09-13T13:50:27.527 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2773189 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: finishing write tid 3 to smithi15938319-18 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: finishing write tid 4 to smithi15938319-18 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: finishing write tid 5 to smithi15938319-18 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: oid 18 updating version 0 to 37 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: oid 18 updating version 37 to 38 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: oid 18 updating version 38 to 39 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:745: oid 18 version 39 is already newer than 36 2024-09-13T13:50:27.538 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 39 (ObjNum 81 snap 4 seq_num 81) dirty exists 2024-09-13T13:50:27.539 INFO:tasks.rados.rados.0.smithi159.stdout:745: left oid 18 (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:27.539 INFO:tasks.rados.rados.0.smithi159.stdout:749: got (0) Success 2024-09-13T13:50:27.539 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T13:50:27.539 INFO:tasks.rados.rados.0.smithi159.stdout:750: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:27.564 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.565 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:27.600 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:27.599+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:27.676 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.677 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.677 INFO:tasks.rados.rados.0.smithi159.stdout:748: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:27.677 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:27.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:27.701 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:27.700+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:27.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:27.756 INFO:tasks.rados.rados.0.smithi159.stdout:746: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-13T13:50:27.757 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:27.787 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:27.787 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:27.796 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:27.826 INFO:tasks.rados.rados.0.smithi159.stdout:751: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:27.826 INFO:tasks.rados.rados.0.smithi159.stdout:751: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: finishing write tid 5 to smithi15938319-14 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: oid 14 updating version 0 to 55 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: oid 14 version 55 is already newer than 53 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: oid 14 updating version 55 to 56 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: oid 14 version 56 is already newer than 54 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 56 (ObjNum 82 snap 4 seq_num 82) dirty exists 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:751: left oid 14 (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:752: finishing write tid 1 to smithi15938319-31 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:745: done (8 left) 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:746: done (7 left) 2024-09-13T13:50:27.827 INFO:tasks.rados.rados.0.smithi159.stdout:747: done (6 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:748: done (5 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:749: done (4 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:750: done (3 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:751: done (2 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:753: done (1 left) 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:754: write (excl) oid 45 current snap is 4 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: finishing write tid 2 to smithi15938319-31 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: finishing write tid 3 to smithi15938319-31 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: finishing write tid 4 to smithi15938319-31 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: finishing write tid 5 to smithi15938319-31 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: oid 31 updating version 0 to 54 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: oid 31 updating version 54 to 55 2024-09-13T13:50:27.828 INFO:tasks.rados.rados.0.smithi159.stdout:752: oid 31 version 55 is already newer than 53 2024-09-13T13:50:27.829 INFO:tasks.rados.rados.0.smithi159.stdout:752: oid 31 version 55 is already newer than 52 2024-09-13T13:50:27.829 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 55 (ObjNum 83 snap 4 seq_num 83) dirty exists 2024-09-13T13:50:27.829 INFO:tasks.rados.rados.0.smithi159.stdout:752: left oid 31 (ObjNum 83 snap 4 seq_num 83) 2024-09-13T13:50:27.829 INFO:tasks.rados.rados.0.smithi159.stdout:754: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-13T13:50:27.829 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:27.825+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:27.832 INFO:tasks.rados.rados.0.smithi159.stdout:754: writing smithi15938319-45 from 628595 to 1104643 tid 1 2024-09-13T13:50:27.840 INFO:tasks.rados.rados.0.smithi159.stdout:754: writing smithi15938319-45 from 1807440 to 2439784 tid 2 2024-09-13T13:50:27.840 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:27.840 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:754: writing smithi15938319-45 from 2934719 to 3387193 tid 3 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:752: done (1 left) 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:755: read oid 36 snap 3 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:755: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:756: tier_evict oid 28 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:756: tier_evict oid 28 snap -1 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:754: finishing write tid 1 to smithi15938319-45 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:754: finishing write tid 2 to smithi15938319-45 2024-09-13T13:50:27.846 INFO:tasks.rados.rados.0.smithi159.stdout:757: read oid 39 snap -1 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:757: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:758: read oid 48 snap -1 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:758: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:759: read oid 21 snap 3 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:759: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:760: tier_evict oid 25 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:760: tier_evict oid 25 snap -1 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:761: read oid 22 snap -1 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:761: expect deleted 2024-09-13T13:50:27.847 INFO:tasks.rados.rados.0.smithi159.stdout:762: set_chunk oid 16 target oid 2024-09-13T13:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:27.864 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:762: set_chunk oid 16 offset: 1816576 length: 229376 target oid 4712ac1a93f071e12ff1b32a1dfe3c2675bb05e410ffdf622a44c5cf807f6179 offset: 0 snap -1 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:763: snap_create 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:760: got (22) Invalid argument 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:756: got (22) Invalid argument 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: finishing write tid 3 to smithi15938319-45 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: finishing write tid 4 to smithi15938319-45 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: finishing write tid 5 to smithi15938319-45 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: oid 45 updating version 0 to 12 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: oid 45 updating version 12 to 14 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: oid 45 version 14 is already newer than 13 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:754: oid 45 updating version 14 to 15 2024-09-13T13:50:27.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 15 (ObjNum 85 snap 4 seq_num 85) dirty exists 2024-09-13T13:50:27.870 INFO:tasks.rados.rados.0.smithi159.stdout:754: left oid 45 (ObjNum 85 snap 4 seq_num 85) 2024-09-13T13:50:27.870 INFO:tasks.rados.rados.0.smithi159.stdout:755: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:27.885 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:27.885 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:27.903 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:27.903+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:27.905 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:27.906 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:27.908 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:27.908 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:27.908 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:27.908 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:27.917 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:27.946 INFO:tasks.rados.rados.0.smithi159.stdout:759: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T13:50:27.947 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 653020 2024-09-13T13:50:27.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:27.947 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:27.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:27.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:27.970 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:27.992 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:28.004 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:28.004+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:28.012 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:28.012 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:28.017 INFO:tasks.rados.rados.0.smithi159.stdout:757: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:28.018 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 235611 2024-09-13T13:50:28.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:28.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:28.052 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.052 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:28.053 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:28.053 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:28.053 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:28.061 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:28.077 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:28.077 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:28.078 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.097 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:28.097 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:28.108 INFO:tasks.rados.rados.0.smithi159.stdout:758: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-13T13:50:28.108 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 472200 2024-09-13T13:50:28.108 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:28.107+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:28.119 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:28.119 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:28.139+0000 7f020e59c640 0 -- 172.21.15.27:0/1384466777 >> v2:172.21.15.122:6808/1255553857 conn(0x7f021021f830 msgr2=0x7f0210221c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:28.146 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:28.166 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.167 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:28.167 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:28.180 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.6. 2024-09-13T13:50:28.180 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:50:28.183 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.449278989, 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.184 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:50:28.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:50:28.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.190 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.449201587999999, 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:50:28.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:50:28.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.449049049999999, 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:50:28.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:50:28.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:50:28.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.448943011000001, 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.203 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:28.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:50:28.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:50:28.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.440603933, 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:50:28.209 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:50:28.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:207 3.2 e41/33 3:755d2207:::smithi15938319-8 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.435848685, 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022139559, 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.214 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745616+0000", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5199999999999996e-07 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751147+0000", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055306139999999997 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751393+0000", 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024623899999999998 2024-09-13T13:50:28.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751399+0000", 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6490000000000002e-06 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751448+0000", 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9016e-05 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751472+0000", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3175000000000001e-05 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767716+0000", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016244341999999998 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767755+0000", 2024-09-13T13:50:28.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8772000000000001e-05 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.413237222999999, 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.219 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:50:28.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:50:28.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.225 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.411470625, 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:50:28.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.406591283000001, 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:50:28.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:50:28.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:50:28.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:50:28.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:219 3.2 e41/33 3:650cb820:::smithi15938319-20 low_tier:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.399500299, 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02650433, 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781965+0000", 2024-09-13T13:50:28.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.452e-06 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789891+0000", 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079261269999999998 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789958+0000", 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6587999999999993e-05 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789965+0000", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790010+0000", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4592000000000003e-05 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790030+0000", 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0630000000000001e-05 2024-09-13T13:50:28.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808419+0000", 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018388315999999998 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808468+0000", 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9168000000000001e-05 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.396079558, 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.243 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:50:28.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:50:28.248 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:227 3.2 e41/33 3:626302dd:::smithi15938319-28 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.391568072, 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024242393000000001, 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789897+0000", 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8999999999999995e-07 2024-09-13T13:50:28.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795632+0000", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057353760000000004 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795680+0000", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8420000000000001e-05 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795692+0000", 2024-09-13T13:50:28.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1452e-05 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795737+0000", 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5596000000000002e-05 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795751+0000", 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3127e-05 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814104+0000", 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018353103999999999 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814138+0000", 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4527999999999999e-05 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:233 3.0 3:3563ec55:::smithi15938319-34 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.385537329, 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020873143, 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 233 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795928+0000", 2024-09-13T13:50:28.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899999999999999e-06 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795935+0000", 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795945+0000", 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0378e-05 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795953+0000", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8509999999999999e-06 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798390+0000", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024377449999999998 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798511+0000", 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120852 2024-09-13T13:50:28.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.805311+0000", 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067998520000000003 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816555+0000", 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011243510999999999 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816742+0000", 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018714599999999999 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816766+0000", 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4450000000000001e-05 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816800+0000", 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3405000000000003e-05 2024-09-13T13:50:28.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.371822268000001, 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.260 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:28.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:50:28.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.368937223, 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.266 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:50:28.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.272 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.366733198, 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:28.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:50:28.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.278 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.359742554, 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:50:28.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:245 3.2 e41/33 3:60c9a0cb:::smithi15938319-46 low_tier:head v 41'9, mlcod=41'6)", 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.354055567, 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024215621999999999, 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827409+0000", 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832412+0000", 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050024800000000001 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832526+0000", 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113959 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832535+0000", 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0850000000000003e-06 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832620+0000", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5050999999999996e-05 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832644+0000", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4012e-05 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851590+0000", 2024-09-13T13:50:28.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018946647 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851624+0000", 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3628999999999999e-05 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:243 3.2 e41/33 3:6893a2ac:::smithi15938319-44 low_tier:head v 41'10, mlcod=41'6)", 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:28.288 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.349046606, 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019400412999999998, 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832418+0000", 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837463+0000", 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050444909999999999 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837489+0000", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6551e-05 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837498+0000", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5920000000000001e-06 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839367+0000", 2024-09-13T13:50:28.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018689220000000001 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839391+0000", 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4811e-05 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851794+0000", 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012402827999999999 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851818+0000", 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3459999999999999e-05 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.254750939999999, 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.293 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:50:28.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:50:28.298 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.299 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.299 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.299 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.299 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.301 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-13T13:50:28.301 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:28.303 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:28.299+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:754: done (9 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:755: done (8 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:756: done (7 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:757: done (6 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:758: done (5 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:759: done (4 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:760: done (3 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:761: done (2 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:763: done (1 left) 2024-09-13T13:50:28.307 INFO:tasks.rados.rados.0.smithi159.stdout:764: write oid 45 current snap is 5 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:762: finishing set_chunk to oid 16 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 1816576 length: 229376 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:764: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:764: writing smithi15938319-45 from 645402 to 1196699 tid 1 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:764: writing smithi15938319-45 from 1749569 to 2241924 tid 2 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:764: writing smithi15938319-45 from 2845789 to 3445402 tid 3 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:762: done (1 left) 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:765: read oid 30 snap -1 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:765: expect deleted 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:766: set_chunk oid 16 target oid 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:766: set_chunk oid 16 offset: 396800 length: 697344 target oid 371245df4d3aba7bcfb292a756fdacb6fe4f3a14213baf69deb042b6d1a0ae6c offset: 0 snap -1 2024-09-13T13:50:28.308 INFO:tasks.rados.rados.0.smithi159.stdout:767: read oid 37 snap 4 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:767: expect deleted 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:768: read oid 12 snap -1 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:768: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:769: read oid 15 snap -1 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:769: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:770: read oid 46 snap 3 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:770: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:771: tier_promote oid 20 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:764: finishing write tid 1 to smithi15938319-45 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:772: tier_promote oid 32 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:764: finishing write tid 2 to smithi15938319-45 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:764: finishing write tid 3 to smithi15938319-45 2024-09-13T13:50:28.309 INFO:tasks.rados.rados.0.smithi159.stdout:764: finishing write tid 4 to smithi15938319-45 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:765: done (8 left) 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:773: read oid 33 snap -1 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:773: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:774: write oid 29 current snap is 5 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: finishing write tid 5 to smithi15938319-45 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: oid 45 updating version 0 to 17 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: oid 45 updating version 17 to 19 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: oid 45 updating version 19 to 20 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: oid 45 version 20 is already newer than 18 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 20 (ObjNum 86 snap 5 seq_num 86) dirty exists 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:764: left oid 45 (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:774: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-13T13:50:28.310 INFO:tasks.rados.rados.0.smithi159.stdout:774: writing smithi15938319-29 from 662209 to 1288755 tid 1 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:774: writing smithi15938319-29 from 1808051 to 2444064 tid 2 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:774: writing smithi15938319-29 from 2873212 to 3462209 tid 3 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:764: done (9 left) 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:775: read oid 35 snap -1 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:775: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:776: write oid 26 current snap is 5 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:766: finishing set_chunk to oid 16 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 396800 length: 697344 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:776: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:776: writing smithi15938319-26 from 679016 to 1380811 tid 1 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:776: writing smithi15938319-26 from 1866533 to 2362557 tid 2 2024-09-13T13:50:28.311 INFO:tasks.rados.rados.0.smithi159.stdout:776: writing smithi15938319-26 from 2900635 to 3303278 tid 3 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:766: done (10 left) 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:777: read oid 39 snap -1 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:777: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:778: write (excl) oid 19 current snap is 5 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:778: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:778: writing smithi15938319-19 from 695823 to 1472867 tid 1 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:778: writing smithi15938319-19 from 1925015 to 2564697 tid 2 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:778: writing smithi15938319-19 from 3328058 to 3495823 tid 3 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:779: read oid 14 snap -1 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:779: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:28.312 INFO:tasks.rados.rados.0.smithi159.stdout:770: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:28.368 INFO:tasks.rados.rados.0.smithi159.stdout:771: got (0) Success 2024-09-13T13:50:28.368 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 28 (ObjNum 77 snap 3 seq_num 77) dirty exists 2024-09-13T13:50:28.368 INFO:tasks.rados.rados.0.smithi159.stdout:774: finishing write tid 1 to smithi15938319-29 2024-09-13T13:50:28.368 INFO:tasks.rados.rados.0.smithi159.stdout:772: got (0) Success 2024-09-13T13:50:28.368 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 19 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:774: finishing write tid 2 to smithi15938319-29 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:770: done (12 left) 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:771: done (11 left) 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:772: done (10 left) 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:780: read oid 40 snap -1 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:780: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:781: read oid 1 snap 3 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:781: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:782: read oid 17 snap -1 2024-09-13T13:50:28.369 INFO:tasks.rados.rados.0.smithi159.stdout:782: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:28.372 INFO:tasks.rados.rados.0.smithi159.stdout:783: write oid 42 current snap is 5 2024-09-13T13:50:28.372 INFO:tasks.rados.rados.0.smithi159.stdout:783: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-13T13:50:28.374 INFO:tasks.rados.rados.0.smithi159.stdout:783: writing smithi15938319-42 from 712630 to 1164923 tid 1 2024-09-13T13:50:28.379 INFO:tasks.rados.rados.0.smithi159.stdout:783: writing smithi15938319-42 from 1583497 to 2083190 tid 2 2024-09-13T13:50:28.384 INFO:tasks.rados.rados.0.smithi159.stdout:783: writing smithi15938319-42 from 2555481 to 2980668 tid 3 2024-09-13T13:50:28.384 INFO:tasks.rados.rados.0.smithi159.stdout:784: set_chunk oid 6 target oid 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:784: set_chunk oid 6 offset: 1136640 length: 195072 target oid f33a904c0f55ae91c087cfd80af249c74c2521dd2ef8cd5953887444ce410098 offset: 0 snap 3 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:785: write oid 23 current snap is 5 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:774: finishing write tid 3 to smithi15938319-29 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:774: finishing write tid 4 to smithi15938319-29 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:774: finishing write tid 5 to smithi15938319-29 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:774: oid 29 updating version 0 to 55 2024-09-13T13:50:28.388 INFO:tasks.rados.rados.0.smithi159.stdout:774: oid 29 updating version 55 to 56 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:774: oid 29 version 56 is already newer than 54 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:774: oid 29 updating version 56 to 57 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 57 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:774: left oid 29 (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:775: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:28.389 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.401 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:28.401+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:28.417 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.450 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 649379 2024-09-13T13:50:28.451 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.451 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:28.451 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:28.451 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:28.471 INFO:tasks.rados.rados.0.smithi159.stdout:776: finishing write tid 1 to smithi15938319-26 2024-09-13T13:50:28.471 INFO:tasks.rados.rados.0.smithi159.stdout:776: finishing write tid 2 to smithi15938319-26 2024-09-13T13:50:28.471 INFO:tasks.rados.rados.0.smithi159.stdout:776: finishing write tid 3 to smithi15938319-26 2024-09-13T13:50:28.471 INFO:tasks.rados.rados.0.smithi159.stdout:776: finishing write tid 4 to smithi15938319-26 2024-09-13T13:50:28.471 INFO:tasks.rados.rados.0.smithi159.stdout:776: finishing write tid 5 to smithi15938319-26 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:776: oid 26 updating version 0 to 36 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:776: oid 26 updating version 36 to 37 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:776: oid 26 version 37 is already newer than 35 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:776: oid 26 updating version 37 to 38 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 38 (ObjNum 88 snap 5 seq_num 88) dirty exists 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:776: left oid 26 (ObjNum 88 snap 5 seq_num 88) 2024-09-13T13:50:28.472 INFO:tasks.rados.rados.0.smithi159.stdout:777: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:28.502 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.502 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:28.501+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:28.532 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.562 INFO:tasks.rados.rados.0.smithi159.stdout:780: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:28.567 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 904971 2024-09-13T13:50:28.577 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.577 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:28.577 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:28.577 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:28.577 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.585 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:28.590 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 737493 2024-09-13T13:50:28.603 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:28.603+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:28.611 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.642 INFO:tasks.rados.rados.0.smithi159.stdout:784: finishing set_chunk to oid 6 2024-09-13T13:50:28.642 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 1136640 length: 195072 2024-09-13T13:50:28.642 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T13:50:28.642 INFO:tasks.rados.rados.0.smithi159.stdout:782: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:28.642 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:28.697 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.953157624999999, 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:50:28.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:50:28.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:28.702 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.947576366, 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:50:28.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.944599755, 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:50:28.707 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:50:28.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:50:28.712 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.942625584, 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:28.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:50:28.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:218 3.0 e41/33 3:0f58cd54:::smithi15938319-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'5, mlcod=41'3)", 2024-09-13T13:50:28.717 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.778326+0000", 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.916146505, 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006363512, 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.778326+0000", 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.778326+0000", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.778327+0000", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.079e-06 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.779023+0000", 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069664200000000003 2024-09-13T13:50:28.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.779065+0000", 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1724999999999998e-05 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.779075+0000", 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4420000000000004e-06 2024-09-13T13:50:28.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.779141+0000", 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6387e-05 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.779167+0000", 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6359e-05 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.784632+0000", 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054651680000000003 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.784689+0000", 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6709999999999997e-05 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.908866483000001, 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:50:28.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:50:28.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.903895061, 2024-09-13T13:50:28.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:28.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:50:28.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:50:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.89999377, 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:28.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:50:28.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:233 3.0 e41/33 3:3563ec55:::smithi15938319-34 low_tier:head v 41'9, mlcod=41'4)", 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.805416+0000", 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.889055783, 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059803399999999998, 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.805416+0000", 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.805416+0000", 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.805418+0000", 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.843e-06 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.806192+0000", 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00077347500000000003 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.806241+0000", 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9725000000000002e-05 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.806255+0000", 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3949e-05 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.806322+0000", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6449e-05 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.806341+0000", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9071000000000001e-05 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811366+0000", 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050253559999999999 2024-09-13T13:50:28.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811397+0000", 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0471999999999999e-05 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:230 3.0 e41/33 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'10, mlcod=41'4)", 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814764+0000", 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.879708186, 2024-09-13T13:50:28.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.007122214, 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814764+0000", 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814764+0000", 2024-09-13T13:50:28.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814765+0000", 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0100000000000001e-06 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.815328+0000", 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00056316299999999999 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.815364+0000", 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6106000000000003e-05 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.815372+0000", 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2160000000000006e-06 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.815426+0000", 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3767999999999997e-05 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.815447+0000", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0701000000000001e-05 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821829+0000", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063819549999999999 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821886+0000", 2024-09-13T13:50:28.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7294999999999999e-05 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:238 3.0 e41/33 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'9)", 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.866581942, 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076037040000000002, 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.748 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827891+0000", 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.248e-06 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828642+0000", 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075090499999999995 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828710+0000", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7478999999999995e-05 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828723+0000", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3499999999999999e-05 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828822+0000", 2024-09-13T13:50:28.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9061999999999994e-05 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828853+0000", 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0939999999999999e-05 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835418+0000", 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065648920000000001 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835494+0000", 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5678000000000003e-05 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.859955702000001, 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.753 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:50:28.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:50:28.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.854779860000001, 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:50:28.758 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.778828782, 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:50:28.763 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.769 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.774027887000001, 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.770 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:28.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:50:28.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:325 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:28.776 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.923366312000001, 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072630530000000002, 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.777 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.780 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 325 2024-09-13T13:50:28.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.780 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771107+0000", 2024-09-13T13:50:28.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.252e-06 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771114+0000", 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0119999999999999e-06 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771129+0000", 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5131e-05 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771137+0000", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9570000000000002e-06 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771199+0000", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2007000000000004e-05 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771272+0000", 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3152999999999996e-05 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777134+0000", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005861387 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778167+0000", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010329270000000001 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778274+0000", 2024-09-13T13:50:28.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106827 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778323+0000", 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9941999999999997e-05 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778369+0000", 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5457999999999997e-05 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.619389050000001, 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:50:28.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:50:28.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:50:28.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.791 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.496507897000001, 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.792 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:50:28.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:50:28.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:50:28.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.43952756399999998, 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.798 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:28.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:50:28.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:50:28.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.43421471499999997, 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:28.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:28.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:50:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:50:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:28.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:28.810 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.811 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:28.725+0000 7ffaeece8640 -1 osd.0 52 handle_osd_map injecting map crc failure 2024-09-13T13:50:28.811 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:28.719+0000 7f8f5ac65640 -1 osd.6 52 handle_osd_map injecting map crc failure 2024-09-13T13:50:28.812 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:28.813 INFO:tasks.rados.rados.0.smithi159.stdout:785: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:785: writing smithi15938319-23 from 729437 to 1256979 tid 1 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:785: writing smithi15938319-23 from 1925626 to 2568977 tid 2 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:785: writing smithi15938319-23 from 3266551 to 3529437 tid 3 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:774: done (15 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:775: done (14 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:776: done (13 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:777: done (12 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:780: done (11 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:782: done (10 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:784: done (9 left) 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:786: tier_promote oid 40 2024-09-13T13:50:28.814 INFO:tasks.rados.rados.0.smithi159.stdout:768: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:28.815 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:28.811+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:28.835 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:28.885 INFO:tasks.rados.rados.0.smithi159.stdout:769: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:28.912 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:28.912+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:28.939 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:28.939 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:28.940 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:28.940 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:28.948 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:28.954 INFO:tasks.rados.rados.0.smithi159.stdout:786: got (0) Success 2024-09-13T13:50:28.954 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T13:50:28.954 INFO:tasks.rados.rados.0.smithi159.stdout:779: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:28.954 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:29.013 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:29.012+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:29.036 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2756562 2024-09-13T13:50:29.047 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:29.051 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:29.052 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:29.052 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:29.062 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: finishing write tid 1 to smithi15938319-42 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: finishing write tid 2 to smithi15938319-42 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: finishing write tid 3 to smithi15938319-42 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: finishing write tid 4 to smithi15938319-42 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: finishing write tid 5 to smithi15938319-42 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: oid 42 updating version 0 to 31 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: oid 42 version 31 is already newer than 30 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: oid 42 version 31 is already newer than 29 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: oid 42 version 31 is already newer than 28 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 31 (ObjNum 90 snap 5 seq_num 90) dirty exists 2024-09-13T13:50:29.102 INFO:tasks.rados.rados.0.smithi159.stdout:783: left oid 42 (ObjNum 90 snap 5 seq_num 90) 2024-09-13T13:50:29.103 INFO:tasks.rados.rados.0.smithi159.stdout:773: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:29.105 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1353650 2024-09-13T13:50:29.146 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:29.145+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:29.167 INFO:tasks.rados.rados.0.smithi159.stdout:785: finishing write tid 1 to smithi15938319-23 2024-09-13T13:50:29.167 INFO:tasks.rados.rados.0.smithi159.stdout:785: finishing write tid 2 to smithi15938319-23 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:785: finishing write tid 3 to smithi15938319-23 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:785: finishing write tid 4 to smithi15938319-23 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:767: done (9 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:768: done (8 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:769: done (7 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:773: done (6 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:779: done (5 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:783: done (4 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:786: done (3 left) 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:787: write oid 48 current snap is 5 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:785: finishing write tid 5 to smithi15938319-23 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:785: oid 23 updating version 0 to 41 2024-09-13T13:50:29.168 INFO:tasks.rados.rados.0.smithi159.stdout:785: oid 23 version 41 is already newer than 40 2024-09-13T13:50:29.169 INFO:tasks.rados.rados.0.smithi159.stdout:785: oid 23 version 41 is already newer than 38 2024-09-13T13:50:29.169 INFO:tasks.rados.rados.0.smithi159.stdout:785: oid 23 version 41 is already newer than 39 2024-09-13T13:50:29.169 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 41 (ObjNum 91 snap 5 seq_num 91) dirty exists 2024-09-13T13:50:29.169 INFO:tasks.rados.rados.0.smithi159.stdout:785: left oid 23 (ObjNum 91 snap 5 seq_num 91) 2024-09-13T13:50:29.169 INFO:tasks.rados.rados.0.smithi159.stdout:787: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-13T13:50:29.175 INFO:tasks.rados.rados.0.smithi159.stdout:787: writing smithi15938319-48 from 746244 to 1465388 tid 1 2024-09-13T13:50:29.181 INFO:tasks.rados.rados.0.smithi159.stdout:787: writing smithi15938319-48 from 2100461 to 2603823 tid 2 2024-09-13T13:50:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:787: writing smithi15938319-48 from 3126680 to 3546244 tid 3 2024-09-13T13:50:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:785: done (3 left) 2024-09-13T13:50:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:788: write (excl) oid 32 current snap is 5 2024-09-13T13:50:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:781: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T13:50:29.189 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1323387 2024-09-13T13:50:29.201 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 233422 2024-09-13T13:50:29.209 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:29.209 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:29.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:29.209 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:29.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.946026532000001, 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:50:29.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:50:29.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.945073033, 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:50:29.214 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:50:29.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:51 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.265856+0000", 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.941906253999999, 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001663152, 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.219 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265856+0000", 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265856+0000", 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265856+0000", 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1900000000000001e-07 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265867+0000", 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0047000000000001e-05 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265902+0000", 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5661999999999998e-05 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265910+0000", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265958+0000", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7521000000000001e-05 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.265972+0000", 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4117999999999999e-05 2024-09-13T13:50:29.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.267492+0000", 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001520258 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.267519+0000", 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6475999999999999e-05 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:125 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'4, mlcod=41'2)", 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.483997+0000", 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.723764847999998, 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001763272, 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.483997+0000", 2024-09-13T13:50:29.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.483997+0000", 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.483998+0000", 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7999999999999993e-07 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.484476+0000", 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00047770700000000001 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.484513+0000", 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7255999999999997e-05 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.484522+0000", 2024-09-13T13:50:29.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6090000000000001e-06 2024-09-13T13:50:29.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.484581+0000", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9471e-05 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.484596+0000", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5160999999999999e-05 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.485716+0000", 2024-09-13T13:50:29.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001119552 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.485760+0000", 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4536000000000002e-05 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:29.229 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.718823059000002, 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:29.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:50:29.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:127 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.717904055999998, 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029472949999999999, 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.234 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489859+0000", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8599999999999998e-07 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489864+0000", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1209999999999998e-06 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489892+0000", 2024-09-13T13:50:29.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8059e-05 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489899+0000", 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4429999999999997e-06 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489946+0000", 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7018000000000003e-05 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489965+0000", 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8419000000000001e-05 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492765+0000", 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028008830000000001 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492805+0000", 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9666000000000003e-05 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:177 4.3 e41/34 4:c1ae59bf:::smithi15938319-45:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.656391+0000", 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.551370990999999, 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018722599999999999, 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.656391+0000", 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.656391+0000", 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.656391+0000", 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:50:29.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.657139+0000", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000748305 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.657163+0000", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3887999999999999e-05 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.657170+0000", 2024-09-13T13:50:29.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9179999999999997e-06 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.657220+0000", 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0083000000000001e-05 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.657239+0000", 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8776000000000001e-05 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.658231+0000", 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00099216600000000005 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.658263+0000", 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1865000000000001e-05 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.243 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:178 4.3 e41/34 4:c1ae59bf:::smithi15938319-45:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.662799+0000", 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.544963038999999, 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0016305950000000001, 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.662799+0000", 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.662799+0000", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.662800+0000", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2900000000000003e-07 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.663406+0000", 2024-09-13T13:50:29.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060633799999999999 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.663428+0000", 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1999999999999999e-05 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.663434+0000", 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.545e-06 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.663484+0000", 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9809999999999999e-05 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.663500+0000", 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5590000000000002e-05 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.664401+0000", 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00090156000000000003 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.664429+0000", 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8022999999999999e-05 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:262 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'7, mlcod=41'5)", 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.296107678999999, 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024065520000000002, 2024-09-13T13:50:29.248 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911655+0000", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1790000000000001e-06 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911665+0000", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7059999999999999e-06 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911706+0000", 2024-09-13T13:50:29.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0985000000000001e-05 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911715+0000", 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4319999999999999e-06 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911772+0000", 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7462e-05 2024-09-13T13:50:29.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911801+0000", 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9023e-05 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.914031+0000", 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022304650000000001 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.914061+0000", 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9300000000000001e-05 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.4342367730000003, 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.253 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1026 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'15, mlcod=42'12)", 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.4275503409999999, 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031666730000000001, 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200000000000002e-07 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780461+0000", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024905400000000002 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780488+0000", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6423000000000001e-05 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780497+0000", 2024-09-13T13:50:29.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.918e-06 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780550+0000", 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3195000000000001e-05 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780574+0000", 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3734e-05 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783341+0000", 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027677679999999999 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783378+0000", 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6769e-05 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.262 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1027 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'16, mlcod=42'12)", 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.4267336149999998, 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024073179999999999, 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781029+0000", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999998e-07 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781037+0000", 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9109999999999993e-06 2024-09-13T13:50:29.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781075+0000", 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8087000000000003e-05 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781082+0000", 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5030000000000002e-06 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781131+0000", 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9073999999999998e-05 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781177+0000", 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6338999999999998e-05 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783410+0000", 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022328259999999998 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783436+0000", 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5758e-05 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1141 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'18, mlcod=48'15)", 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:24.984980+0000", 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.2227815919999996, 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001821116, 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.267 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.984980+0000", 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.984980+0000", 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.984981+0000", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4799999999999998e-07 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.985006+0000", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4661999999999999e-05 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.985072+0000", 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5963000000000005e-05 2024-09-13T13:50:29.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.985080+0000", 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1119999999999996e-06 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.985124+0000", 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4471000000000002e-05 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.985138+0000", 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.34e-05 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986762+0000", 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001624037 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986801+0000", 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9823000000000001e-05 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.271 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1142 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'19, mlcod=48'15)", 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:24.986315+0000", 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.2214466550000003, 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00087163099999999999, 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986315+0000", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986315+0000", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986316+0000", 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-09-13T13:50:29.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986323+0000", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2579999999999998e-06 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986354+0000", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0902000000000003e-05 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986361+0000", 2024-09-13T13:50:29.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8290000000000002e-06 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986410+0000", 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9171000000000001e-05 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.986431+0000", 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1010999999999999e-05 2024-09-13T13:50:29.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.987163+0000", 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000731538 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:24.987187+0000", 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4107000000000001e-05 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1216 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'17, mlcod=51'14)", 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.206892+0000", 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 1.000870385, 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002158512, 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.206892+0000", 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.206892+0000", 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.206892+0000", 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.05e-07 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.207361+0000", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046883800000000001 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.207383+0000", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2447e-05 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.207390+0000", 2024-09-13T13:50:29.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1500000000000002e-06 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.207437+0000", 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7210999999999999e-05 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.207455+0000", 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7467000000000001e-05 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.209020+0000", 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0015647530000000001 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.209050+0000", 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0440999999999999e-05 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.281 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1217 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'18, mlcod=51'15)", 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.212094+0000", 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.99566799299999997, 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0014423089999999999, 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212094+0000", 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212094+0000", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212095+0000", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1999999999999999e-07 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212538+0000", 2024-09-13T13:50:29.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044355600000000001 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212559+0000", 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1274999999999999e-05 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212565+0000", 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.203e-06 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212604+0000", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9017000000000003e-05 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.212618+0000", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4535000000000001e-05 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.213510+0000", 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000892014 2024-09-13T13:50:29.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.213536+0000", 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5989000000000001e-05 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.98887099300000003, 2024-09-13T13:50:29.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:29.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:50:29.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1219 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'20, mlcod=52'17)", 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.98801520300000001, 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031764929999999999, 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.291 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219751+0000", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9310000000000001e-06 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219767+0000", 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6053999999999999e-05 2024-09-13T13:50:29.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219771+0000", 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4130000000000002e-06 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219981+0000", 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00021069900000000001 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219994+0000", 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3174e-05 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222896+0000", 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029013910000000001 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222923+0000", 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7382e-05 2024-09-13T13:50:29.295 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:91 4.8 e52/34 4:165e03eb:::smithi15938319-32:head v 52'22, mlcod=48'19)", 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.97102549400000004, 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030596489999999998, 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236737+0000", 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.79e-07 2024-09-13T13:50:29.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237317+0000", 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057988500000000004 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237354+0000", 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7883000000000003e-05 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237362+0000", 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4660000000000002e-06 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237421+0000", 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8588999999999998e-05 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237446+0000", 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5918000000000002e-05 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239751+0000", 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023044540000000001 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239796+0000", 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5175e-05 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "pg_update_log_missing(4.8 epoch 52/34 rep_tid 92 entries 52'23 (0'0) error 4:165e03eb:::smithi15938319-32:head by client.4571.0:1240 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'19)", 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.238644+0000", 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.96911758199999998, 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012786850000000001, 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238644+0000", 2024-09-13T13:50:29.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238644+0000", 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238645+0000", 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7400000000000002e-07 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238656+0000", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1070000000000001e-05 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238701+0000", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4595999999999998e-05 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.238719+0000", 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8576999999999998e-05 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239084+0000", 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00036417600000000002 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239923+0000", 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000839392 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.304 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.305 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:29.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:29.308 INFO:tasks.rados.rados.0.smithi159.stdout:778: finishing write tid 1 to smithi15938319-19 2024-09-13T13:50:29.308 INFO:tasks.rados.rados.0.smithi159.stdout:778: finishing write tid 2 to smithi15938319-19 2024-09-13T13:50:29.308 INFO:tasks.rados.rados.0.smithi159.stdout:778: finishing write tid 3 to smithi15938319-19 2024-09-13T13:50:29.308 INFO:tasks.rados.rados.0.smithi159.stdout:778: finishing write tid 4 to smithi15938319-19 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: finishing write tid 1 to smithi15938319-48 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: finishing write tid 2 to smithi15938319-48 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: finishing write tid 3 to smithi15938319-48 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: finishing write tid 4 to smithi15938319-48 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: finishing write tid 5 to smithi15938319-48 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: oid 48 updating version 0 to 59 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: oid 48 updating version 59 to 61 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: oid 48 version 61 is already newer than 60 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: oid 48 updating version 61 to 62 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 62 (ObjNum 92 snap 5 seq_num 92) dirty exists 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:787: left oid 48 (ObjNum 92 snap 5 seq_num 92) 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:778: finishing write tid 5 to smithi15938319-19 2024-09-13T13:50:29.309 INFO:tasks.rados.rados.0.smithi159.stdout:778: oid 19 updating version 0 to 49 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:778: oid 19 updating version 49 to 51 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:778: oid 19 updating version 51 to 52 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:778: oid 19 version 52 is already newer than 50 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 52 (ObjNum 89 snap 5 seq_num 89) dirty exists 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:778: left oid 19 (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:788: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:788: writing smithi15938319-32 from 763051 to 1557444 tid 1 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:788: writing smithi15938319-32 from 2158943 to 2805963 tid 2 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:788: writing smithi15938319-32 from 3437750 to 3563051 tid 3 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:778: done (3 left) 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:781: done (2 left) 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:787: done (1 left) 2024-09-13T13:50:29.310 INFO:tasks.rados.rados.0.smithi159.stdout:789: read oid 50 snap -1 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:789: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:790: set_chunk oid 49 target oid 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:790: set_chunk oid 49 offset: 2227200 length: 67072 target oid 184cff5cb8d1319539ad87060abbc2771f37118bf2217993d80966f2c6f0b90b offset: 0 snap -1 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:791: snap_create 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: finishing write tid 1 to smithi15938319-32 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: finishing write tid 2 to smithi15938319-32 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: finishing write tid 3 to smithi15938319-32 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: finishing write tid 4 to smithi15938319-32 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: finishing write tid 5 to smithi15938319-32 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: oid 32 updating version 0 to 25 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: oid 32 updating version 25 to 26 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: oid 32 updating version 26 to 28 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:788: oid 32 version 28 is already newer than 27 2024-09-13T13:50:29.311 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 28 (ObjNum 93 snap 5 seq_num 93) dirty exists 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:788: left oid 32 (ObjNum 93 snap 5 seq_num 93) 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:790: finishing set_chunk to oid 49 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 2227200 length: 67072 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:789: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:29.312 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:29.312 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:29.305+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:29.407 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:29.406+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:29.451 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:29.451 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:29.451 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:29.451 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:29.459 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:29.508 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:29.507+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:29.605 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:29.605 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:29.605 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:29.605 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:29.605 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:29.608 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:29.608+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:29.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:29.710 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:29.709+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:29.719 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:29.719+0000 7f3c37938640 -1 osd.5 53 handle_osd_map injecting map crc failure 2024-09-13T13:50:29.722 INFO:tasks.rados.rados.0.smithi159.stdout:788: done (3 left) 2024-09-13T13:50:29.722 INFO:tasks.rados.rados.0.smithi159.stdout:789: done (2 left) 2024-09-13T13:50:29.723 INFO:tasks.rados.rados.0.smithi159.stdout:790: done (1 left) 2024-09-13T13:50:29.723 INFO:tasks.rados.rados.0.smithi159.stdout:791: done (0 left) 2024-09-13T13:50:29.723 INFO:tasks.rados.rados.0.smithi159.stdout:792: write (excl) oid 8 current snap is 6 2024-09-13T13:50:29.723 INFO:tasks.rados.rados.0.smithi159.stdout:792: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-13T13:50:29.729 INFO:tasks.rados.rados.0.smithi159.stdout:792: writing smithi15938319-8 from 779858 to 1249500 tid 1 2024-09-13T13:50:29.735 INFO:tasks.rados.rados.0.smithi159.stdout:792: writing smithi15938319-8 from 1817425 to 2324456 tid 2 2024-09-13T13:50:29.741 INFO:tasks.rados.rados.0.smithi159.stdout:792: writing smithi15938319-8 from 2781526 to 3368154 tid 3 2024-09-13T13:50:29.741 INFO:tasks.rados.rados.0.smithi159.stdout:793: read oid 7 snap -1 2024-09-13T13:50:29.741 INFO:tasks.rados.rados.0.smithi159.stdout:793: expect (ObjNum 80 snap 3 seq_num 80) 2024-09-13T13:50:29.741 INFO:tasks.rados.rados.0.smithi159.stdout:794: read oid 40 snap -1 2024-09-13T13:50:29.742 INFO:tasks.rados.rados.0.smithi159.stdout:794: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:29.742 INFO:tasks.rados.rados.0.smithi159.stdout:795: read oid 12 snap 4 2024-09-13T13:50:29.742 INFO:tasks.rados.rados.0.smithi159.stdout:795: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:29.742 INFO:tasks.rados.rados.0.smithi159.stdout:796: set_chunk oid 21 target oid 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:29.762 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 41.218114765999999, 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:29.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:50:29.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:91 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'65, mlcod=25'60)", 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.558702511999996, 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004785274, 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201427+0000", 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2900000000000002e-07 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201442+0000", 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4711e-05 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201506+0000", 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4028000000000004e-05 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201518+0000", 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.149e-05 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201579+0000", 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0806e-05 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201600+0000", 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1282000000000001e-05 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206182+0000", 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045820169999999999 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206212+0000", 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0111e-05 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.771 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:92 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'66, mlcod=25'60)", 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.558679493, 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047825760000000002, 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201450+0000", 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:50:29.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201455+0000", 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201536+0000", 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0749999999999998e-05 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201539+0000", 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9840000000000001e-06 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205053+0000", 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035134839999999999 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205066+0000", 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3079e-05 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206215+0000", 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011490509999999999 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206232+0000", 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7416999999999999e-05 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.776 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.558672700000002, 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:50:29.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:50:29.781 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.557284476, 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:50:29.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.556582472999999, 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.786 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:50:29.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.790 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:96 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'63)", 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.556096670000002, 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050574289999999996, 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:29.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204033+0000", 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204133+0000", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9450000000000005e-05 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204163+0000", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0018999999999998e-05 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204168+0000", 2024-09-13T13:50:29.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5539999999999997e-06 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206261+0000", 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020930889999999998 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206274+0000", 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3018e-05 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209065+0000", 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027905909999999998 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209090+0000", 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4896999999999999e-05 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:97 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'71, mlcod=25'63)", 2024-09-13T13:50:29.795 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.555488246000003, 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044687770000000002, 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204641+0000", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204642+0000", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204651+0000", 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2539999999999998e-06 2024-09-13T13:50:29.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204708+0000", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6959000000000001e-05 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204719+0000", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0825999999999999e-05 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206607+0000", 2024-09-13T13:50:29.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018877659999999999 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206620+0000", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3434999999999999e-05 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209092+0000", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024711659999999999 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209110+0000", 2024-09-13T13:50:29.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8051e-05 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.162980136000002, 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.800 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.805 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:116 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'86, mlcod=25'80)", 2024-09-13T13:50:29.805 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:29.805 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.162435696999999, 2024-09-13T13:50:29.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051291219999999998, 2024-09-13T13:50:29.808 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597694+0000", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597697+0000", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6440000000000002e-06 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597713+0000", 2024-09-13T13:50:29.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5699999999999999e-05 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597718+0000", 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1669999999999998e-06 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599712+0000", 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019937420000000002 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599730+0000", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8091e-05 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602796+0000", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003066306 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602823+0000", 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6112999999999999e-05 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.812 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.156856333, 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:50:29.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.817 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:122 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'92, mlcod=26'88)", 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.155615595, 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045221840000000003, 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604514+0000", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3500000000000002e-07 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604518+0000", 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7440000000000001e-06 2024-09-13T13:50:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604535+0000", 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7088e-05 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.604539+0000", 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.681e-06 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.605822+0000", 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012838680000000001 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.605837+0000", 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.436e-05 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609005+0000", 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031678019999999999 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609036+0000", 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1205999999999999e-05 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:139 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'105, mlcod=27'100)", 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.712071991000002, 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045833219999999999, 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048057+0000", 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048058+0000", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048062+0000", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7319999999999998e-06 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048079+0000", 2024-09-13T13:50:29.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6572e-05 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048085+0000", 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048124+0000", 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9725999999999999e-05 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.048139+0000", 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4874e-05 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052612+0000", 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044729260000000003 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:52.052640+0000", 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8223999999999999e-05 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.826 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:21 4.d e41/34 4:b771c1ad:::smithi15938319-6:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 17.579656447000001, 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049861549999999999, 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180474+0000", 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5799999999999998e-07 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181463+0000", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009894490000000001 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181490+0000", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7053999999999999e-05 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181498+0000", 2024-09-13T13:50:29.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8460000000000008e-06 2024-09-13T13:50:29.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181554+0000", 2024-09-13T13:50:29.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6335000000000002e-05 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181571+0000", 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6328e-05 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.185401+0000", 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038303349999999998 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.185459+0000", 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7849999999999997e-05 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.843299176999999, 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:50:29.832 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:50:29.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:29.838 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.989460428999999, 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:50:29.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:812 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 2412870~427480,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 11.570110162000001, 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054419789999999996, 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 812 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.845 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190022+0000", 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190034+0000", 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190058+0000", 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190071+0000", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3861e-05 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190156+0000", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4179999999999997e-05 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190164+0000", 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-09-13T13:50:29.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194705+0000", 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045405109999999997 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194761+0000", 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6140999999999997e-05 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195461+0000", 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070009399999999996 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 11.570025559999999, 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.850 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:50:29.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:29.855 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 11.56997761, 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:50:29.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:50:29.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.860 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.8815146830000007, 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:29.861 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:50:29.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:29.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:50:29.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:29.867 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:29.868 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:29.867+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:29.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:29.930 INFO:tasks.rados.rados.0.smithi159.stdout:796: set_chunk oid 21 offset: 557056 length: 1190400 target oid f5d860de9d23001e8f1036bcadf5f1335a4c9f6a2e5aff60d31d699ac703778b offset: 0 snap -1 2024-09-13T13:50:29.931 INFO:tasks.rados.rados.0.smithi159.stdout:797: set_chunk oid 2 target oid 2024-09-13T13:50:29.942 INFO:tasks.rados.rados.0.smithi159.stdout:797: set_chunk oid 2 offset: 116736 length: 1314816 target oid a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc offset: 0 snap -1 2024-09-13T13:50:29.942 INFO:tasks.rados.rados.0.smithi159.stdout:798: write oid 25 current snap is 6 2024-09-13T13:50:29.942 INFO:tasks.rados.rados.0.smithi159.stdout:792: finishing write tid 1 to smithi15938319-8 2024-09-13T13:50:29.942 INFO:tasks.rados.rados.0.smithi159.stdout:798: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-13T13:50:29.949 INFO:tasks.rados.rados.0.smithi159.stdout:798: writing smithi15938319-25 from 796665 to 1341556 tid 1 2024-09-13T13:50:29.955 INFO:tasks.rados.rados.0.smithi159.stdout:798: writing smithi15938319-25 from 1759554 to 2410243 tid 2 2024-09-13T13:50:29.961 INFO:tasks.rados.rados.0.smithi159.stdout:798: writing smithi15938319-25 from 2976243 to 3596665 tid 3 2024-09-13T13:50:29.961 INFO:tasks.rados.rados.0.smithi159.stdout:799: tier_evict oid 41 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:799: tier_evict oid 41 snap -1 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: finishing write tid 2 to smithi15938319-8 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:800: tier_flush oid 16 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:800: tier_flush oid 16 snap -1 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: finishing write tid 3 to smithi15938319-8 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: finishing write tid 4 to smithi15938319-8 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: finishing write tid 5 to smithi15938319-8 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: oid 8 updating version 0 to 35 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: oid 8 version 35 is already newer than 33 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: oid 8 version 35 is already newer than 34 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: oid 8 updating version 35 to 36 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 36 (ObjNum 94 snap 6 seq_num 94) dirty exists 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:792: left oid 8 (ObjNum 94 snap 6 seq_num 94) 2024-09-13T13:50:29.962 INFO:tasks.rados.rados.0.smithi159.stdout:795: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:29.963 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:29.961+0000 7fe0fbe07640 -1 osd.2 53 handle_osd_map injecting map crc failure 2024-09-13T13:50:29.969 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:29.969+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:29.977 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:29.976+0000 7faeaa210640 -1 osd.8 53 handle_osd_map injecting map crc failure 2024-09-13T13:50:29.991 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:29.991 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:29.991 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:29.991 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:30.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:30.064 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:30.070 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:30.069+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:30.108 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:30.109 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:30.109 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:30.109 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:30.109 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:30.111 INFO:tasks.rados.rados.0.smithi159.stdout:794: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:30.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:30.171 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:30.169+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:30.177 INFO:tasks.rados.rados.0.smithi159.stdout:793: expect (ObjNum 80 snap 3 seq_num 80) 2024-09-13T13:50:30.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:a1652551:::smithi15938319-33:head, version: 45'26, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@45'26, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.227 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7665654210000001, 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020046090000000001, 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458868+0000", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5199999999999997e-07 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459925+0000", 2024-09-13T13:50:30.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010574549999999999 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459958+0000", 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2385999999999998e-05 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459973+0000", 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5307000000000001e-05 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460061+0000", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7593e-05 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460077+0000", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6286e-05 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460872+0000", 2024-09-13T13:50:30.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00079502999999999996 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(1.0 48/47 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 27'111, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@27'111, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.461711+0000", 2024-09-13T13:50:30.232 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7637217510000003, 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00091073800000000002, 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.461711+0000", 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.461711+0000", 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.461712+0000", 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462019+0000", 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030726499999999998 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462045+0000", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6434000000000001e-05 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462057+0000", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1515000000000001e-05 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462146+0000", 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9411000000000002e-05 2024-09-13T13:50:30.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462160+0000", 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3998e-05 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462622+0000", 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046165799999999999 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:a9f43e9a:::smithi15938319-36:head, version: 42'20, data_included: [585728~701644,1863680~346932], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a9f43e9a:::smithi15938319-36:head@42'20, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2210612, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.462586+0000", 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7628467800000003, 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016267810000000001, 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462586+0000", 2024-09-13T13:50:30.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462586+0000", 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.462587+0000", 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7599999999999997e-07 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463553+0000", 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00096644799999999996 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463581+0000", 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7841999999999999e-05 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463597+0000", 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6319000000000001e-05 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463658+0000", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0940000000000003e-05 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463667+0000", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.653e-06 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464213+0000", 2024-09-13T13:50:30.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054590300000000003 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:a1652551:::smithi15938319-33:head, version: 45'26, data_included: [1269760~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@45'26, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2318336, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.463685+0000", 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7617482610000001, 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018727640000000001, 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.241 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463685+0000", 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463685+0000", 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.463685+0000", 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6400000000000003e-07 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464950+0000", 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012644959999999999 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464996+0000", 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6882999999999998e-05 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465020+0000", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3085999999999999e-05 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465090+0000", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0770000000000002e-05 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465098+0000", 2024-09-13T13:50:30.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8450000000000003e-06 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465557+0000", 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00045921999999999998 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(1.0 48/47 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 27'112, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@27'112, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.464279+0000", 2024-09-13T13:50:30.245 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7611543709999999, 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013444469999999999, 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464279+0000", 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464279+0000", 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464279+0000", 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4400000000000001e-07 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464918+0000", 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063855699999999999 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464952+0000", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4501000000000001e-05 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.464964+0000", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2398999999999999e-05 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465061+0000", 2024-09-13T13:50:30.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6821e-05 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465075+0000", 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4238e-05 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465623+0000", 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054748700000000004 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.249 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:a9f43e9a:::smithi15938319-36:head, version: 42'20, data_included: [2210612~144171], data_size: 144171, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a9f43e9a:::smithi15938319-36:head@42'20, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2588245, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2210612, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.465538+0000", 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7598945439999998, 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015437230000000001, 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465538+0000", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465538+0000", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465539+0000", 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7599999999999997e-07 2024-09-13T13:50:30.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465718+0000", 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017856799999999999 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465755+0000", 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7571000000000003e-05 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465772+0000", 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6461999999999999e-05 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465846+0000", 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3887999999999995e-05 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.465854+0000", 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3149999999999992e-06 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.467082+0000", 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001228243 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:aebdca62:::smithi15938319-23:head, version: 42'21, data_included: [765952~691986,2019328~350426], data_size: 1042412, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aebdca62:::smithi15938319-23:head@42'21, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2369754, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.469551+0000", 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7558817759999998, 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0017095229999999999, 2024-09-13T13:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.469551+0000", 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.469551+0000", 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.469552+0000", 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0500000000000004e-07 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.470529+0000", 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00097725299999999998 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.470569+0000", 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0170000000000003e-05 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.470583+0000", 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4401e-05 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.470662+0000", 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8373000000000003e-05 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.470672+0000", 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6860000000000001e-06 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.471261+0000", 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058913499999999996 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:aae6a8d6:::smithi15938319-43:head, version: 42'25, data_included: [471040~777077,1757184~271499], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aae6a8d6:::smithi15938319-43:head@42'25, size: 2873964, copy_subset: [(0, 2873964)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2028683, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7464742409999996, 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023915640000000001, 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478973+0000", 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4603000000000001e-05 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480619+0000", 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016456089999999999 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480652+0000", 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3485999999999998e-05 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480672+0000", 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9434999999999999e-05 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480788+0000", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116295 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480798+0000", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0159e-05 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.481350+0000", 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055197700000000004 2024-09-13T13:50:30.262 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:aae6a8d6:::smithi15938319-43:head, version: 42'25, data_included: [2028683~373786,2777088~96876], data_size: 470662, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:aae6a8d6:::smithi15938319-43:head@42'25, size: 2873964, copy_subset: [(0, 2873964)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2873964, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2028683, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.483281+0000", 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 6.7421523030000001, 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001251643, 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483281+0000", 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483281+0000", 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483281+0000", 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2500000000000005e-07 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483707+0000", 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00042568800000000002 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483738+0000", 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0830999999999997e-05 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483749+0000", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0936e-05 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483818+0000", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9548000000000006e-05 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.483826+0000", 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0229999999999992e-06 2024-09-13T13:50:30.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.484532+0000", 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00070599200000000001 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1070 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'29, mlcod=45'27)", 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8843649209999995, 2024-09-13T13:50:30.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025038590000000002, 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:30.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341069+0000", 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1200000000000001e-06 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341704+0000", 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063504999999999996 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341739+0000", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.451e-05 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341746+0000", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4279999999999999e-06 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341808+0000", 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1482000000000005e-05 2024-09-13T13:50:30.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341825+0000", 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7187e-05 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.343519+0000", 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016944869999999999 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.343572+0000", 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2595e-05 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1071 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'30, mlcod=45'27)", 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.345895+0000", 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8795376020000001, 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015365050000000001, 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.272 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.345895+0000", 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.345895+0000", 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.345896+0000", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4700000000000001e-07 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.346280+0000", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00038385099999999999 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.346304+0000", 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3725000000000001e-05 2024-09-13T13:50:30.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.346309+0000", 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4049999999999999e-06 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.346356+0000", 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7382999999999997e-05 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.346373+0000", 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6359e-05 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.347388+0000", 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010151400000000001 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.347432+0000", 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3894999999999999e-05 2024-09-13T13:50:30.276 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1072 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'31, mlcod=48'29)", 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8729777209999998, 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001961282, 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352456+0000", 2024-09-13T13:50:30.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.16e-07 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353068+0000", 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00061185999999999999 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353099+0000", 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1364e-05 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353105+0000", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2230000000000004e-06 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353157+0000", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1180999999999997e-05 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353175+0000", 2024-09-13T13:50:30.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8136999999999999e-05 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.354365+0000", 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011900859999999999 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.354416+0000", 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1615e-05 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.281 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8720190270000003, 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:50:30.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2325668460000001, 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:30.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:50:30.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:30.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:50:30.291 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.2299698890000004, 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.294 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:30.295 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.490851259, 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022343459999999999, 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.296 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8800000000000002e-07 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734597+0000", 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4083000000000001e-05 2024-09-13T13:50:30.297 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734630+0000", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3966e-05 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734638+0000", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4919999999999996e-06 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734686+0000", 2024-09-13T13:50:30.298 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8118000000000002e-05 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736816+0000", 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021298990000000002 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:30.299 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.490260146, 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.300 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.301 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:50:30.302 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.747713+0000", 2024-09-13T13:50:30.303 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.4777199699999999, 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011329179999999999, 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747713+0000", 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.304 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747713+0000", 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747714+0000", 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4199999999999995e-07 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747724+0000", 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0263e-05 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747748+0000", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3866000000000002e-05 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747753+0000", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7570000000000001e-06 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747799+0000", 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5241999999999998e-05 2024-09-13T13:50:30.306 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748846+0000", 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010471479999999999 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.4774776940000001, 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022820639999999999, 2024-09-13T13:50:30.307 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:30.308 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747956+0000", 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.63e-07 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747966+0000", 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0353999999999999e-05 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.309 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747991+0000", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4896e-05 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747997+0000", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8730000000000004e-06 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748883+0000", 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088666999999999999 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.310 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750237+0000", 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001353808 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 1.476815421, 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:50:30.311 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:30.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:50:30.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:30.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:30.315 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:30.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:50:30.317 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1279090 2024-09-13T13:50:30.317 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:30.316+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:30.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:30.318 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:30.319 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:30.321 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2713556 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:796: finishing set_chunk to oid 21 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 557056 length: 1190400 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:797: finishing set_chunk to oid 2 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 116736 length: 1314816 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:798: finishing write tid 1 to smithi15938319-25 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:798: finishing write tid 2 to smithi15938319-25 2024-09-13T13:50:30.337 INFO:tasks.rados.rados.0.smithi159.stdout:799: got (2) No such file or directory 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:799: got expected ENOENT (src dne) 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: finishing write tid 3 to smithi15938319-25 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: finishing write tid 4 to smithi15938319-25 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: finishing write tid 5 to smithi15938319-25 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: oid 25 updating version 0 to 57 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: oid 25 version 57 is already newer than 56 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: oid 25 version 57 is already newer than 55 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: oid 25 version 57 is already newer than 54 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 57 (ObjNum 95 snap 6 seq_num 95) dirty exists 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:798: left oid 25 (ObjNum 95 snap 6 seq_num 95) 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:800: got (0) Success 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:792: done (8 left) 2024-09-13T13:50:30.338 INFO:tasks.rados.rados.0.smithi159.stdout:793: done (7 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:794: done (6 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:795: done (5 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:796: done (4 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:797: done (3 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:798: done (2 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:799: done (1 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:800: done (0 left) 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:801: tier_evict oid 50 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:801: tier_evict oid 50 snap -1 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:802: write (excl) oid 33 current snap is 6 2024-09-13T13:50:30.339 INFO:tasks.rados.rados.0.smithi159.stdout:802: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-13T13:50:30.340 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:30.340 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:30.341 INFO:tasks.rados.rados.0.smithi159.stdout:802: writing smithi15938319-33 from 413472 to 1033612 tid 1 2024-09-13T13:50:30.348 INFO:tasks.rados.rados.0.smithi159.stdout:802: writing smithi15938319-33 from 1818036 to 2328736 tid 2 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:802: writing smithi15938319-33 from 3120019 to 3613472 tid 3 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:803: read oid 37 snap -1 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:803: expect deleted 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:804: tier_evict oid 4 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:804: tier_evict oid 4 snap -1 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:801: got (22) Invalid argument 2024-09-13T13:50:30.356 INFO:tasks.rados.rados.0.smithi159.stdout:802: finishing write tid 1 to smithi15938319-33 2024-09-13T13:50:30.357 INFO:tasks.rados.rados.0.smithi159.stdout:802: finishing write tid 2 to smithi15938319-33 2024-09-13T13:50:30.357 INFO:tasks.rados.rados.0.smithi159.stdout:801: done (3 left) 2024-09-13T13:50:30.357 INFO:tasks.rados.rados.0.smithi159.stdout:805: write oid 25 current snap is 6 2024-09-13T13:50:30.357 INFO:tasks.rados.rados.0.smithi159.stdout:805: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-13T13:50:30.361 INFO:tasks.rados.rados.0.smithi159.stdout:805: writing smithi15938319-25 from 430279 to 1125668 tid 1 2024-09-13T13:50:30.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:30.369 INFO:tasks.rados.rados.0.smithi159.stdout:805: writing smithi15938319-25 from 1876518 to 2530876 tid 2 2024-09-13T13:50:30.375 INFO:tasks.rados.rados.0.smithi159.stdout:805: writing smithi15938319-25 from 3031089 to 3567886 tid 3 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:806: read oid 42 snap 5 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:806: expect (ObjNum 90 snap 5 seq_num 90) 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:807: tier_promote oid 38 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:804: got (22) Invalid argument 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:802: finishing write tid 3 to smithi15938319-33 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:802: finishing write tid 4 to smithi15938319-33 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:803: done (5 left) 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:804: done (4 left) 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:808: tier_flush oid 22 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:808: tier_flush oid 22 snap -1 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:802: finishing write tid 5 to smithi15938319-33 2024-09-13T13:50:30.376 INFO:tasks.rados.rados.0.smithi159.stdout:802: oid 33 updating version 0 to 46 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:802: oid 33 version 46 is already newer than 43 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:802: oid 33 version 46 is already newer than 44 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:802: oid 33 version 46 is already newer than 45 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 46 (ObjNum 96 snap 6 seq_num 96) dirty exists 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:802: left oid 33 (ObjNum 96 snap 6 seq_num 96) 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:805: finishing write tid 1 to smithi15938319-25 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:805: finishing write tid 2 to smithi15938319-25 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:802: done (4 left) 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:809: read oid 6 snap 4 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:809: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T13:50:30.377 INFO:tasks.rados.rados.0.smithi159.stdout:810: set_chunk oid 8 target oid 2024-09-13T13:50:30.386 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:30.386 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:30.406 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:30.407 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:30.414 INFO:tasks.rados.rados.0.smithi159.stdout:810: set_chunk oid 8 offset: 376832 length: 2472960 target oid 97c4a3fba2cd96841c0ec7ed72b6931bc8aa4befc084854973245f56bbe2e916 offset: 0 snap -1 2024-09-13T13:50:30.414 INFO:tasks.rados.rados.0.smithi159.stdout:811: set_chunk oid 36 target oid 2024-09-13T13:50:30.417 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:30.416+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:811: set_chunk oid 36 offset: 748544 length: 554496 target oid 1a45436b377aaa4d4b955fa8f76b1f67942117a2c69c555900361712ff923471 offset: 0 snap 3 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:812: read oid 18 snap 3 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:812: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:808: got (2) No such file or directory 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:808: got expected ENOENT (src dne) 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:813: read oid 12 snap 3 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:813: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:30.423 INFO:tasks.rados.rados.0.smithi159.stdout:808: done (8 left) 2024-09-13T13:50:30.424 INFO:tasks.rados.rados.0.smithi159.stdout:814: read oid 46 snap -1 2024-09-13T13:50:30.424 INFO:tasks.rados.rados.0.smithi159.stdout:814: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:30.424 INFO:tasks.rados.rados.0.smithi159.stdout:815: tier_evict oid 40 2024-09-13T13:50:30.424 INFO:tasks.rados.rados.0.smithi159.stdout:815: tier_evict oid 40 snap -1 2024-09-13T13:50:30.424 INFO:tasks.rados.rados.0.smithi159.stdout:809: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T13:50:30.432 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:30.452 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1123687 2024-09-13T13:50:30.452 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:30.453 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:30.454 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:30.454 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:30.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:30.477 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: finishing write tid 3 to smithi15938319-25 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: finishing write tid 4 to smithi15938319-25 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: finishing write tid 5 to smithi15938319-25 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: oid 25 updating version 0 to 58 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: oid 25 updating version 58 to 60 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: oid 25 updating version 60 to 61 2024-09-13T13:50:30.489 INFO:tasks.rados.rados.0.smithi159.stdout:805: oid 25 version 61 is already newer than 59 2024-09-13T13:50:30.490 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 61 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:50:30.490 INFO:tasks.rados.rados.0.smithi159.stdout:805: left oid 25 (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:50:30.490 INFO:tasks.rados.rados.0.smithi159.stdout:807: got (0) Success 2024-09-13T13:50:30.490 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:30.490 INFO:tasks.rados.rados.0.smithi159.stdout:806: expect (ObjNum 90 snap 5 seq_num 90) 2024-09-13T13:50:30.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:30.519 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:30.518+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:30.521 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:30.521 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:30.542 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:30.543 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:30.543 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:30.564 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:30.564 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:30.585 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:30.585 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:30.585 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:30.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:30.607 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:30.610+0000 7ff359d9b640 0 -- 172.21.15.27:0/2023011673 >> v2:172.21.15.122:6808/1255553857 conn(0x7ff35c2201c0 msgr2=0x7ff35c2225b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:30.619 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:30.619+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:30.635 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:30.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:30.656 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:30.656 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:30.675 INFO:tasks.rados.rados.0.smithi159.stdout:810: finishing set_chunk to oid 8 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 36 (ObjNum 94 snap 6 seq_num 94) dirty exists 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:811: finishing set_chunk to oid 36 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 748544 length: 554496 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:815: got (22) Invalid argument 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:814: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:30.721 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:30.720+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:30.721 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:50:30.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:30.738 INFO:tasks.rados.rados.0.smithi159.stdout:812: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T13:50:30.745 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1175249 2024-09-13T13:50:30.756 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:30.781 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:30.805 INFO:tasks.rados.rados.0.smithi159.stdout:813: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:30.811 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1021661 2024-09-13T13:50:30.821 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:30.820+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:30.833 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:30.833 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:30.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:30.847 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 333258 2024-09-13T13:50:30.910 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1985863 2024-09-13T13:50:30.922 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:30.920+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:30.929 INFO:tasks.rados.rados.0.smithi159.stdout:805: done (9 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:806: done (8 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:807: done (7 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:809: done (6 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:810: done (5 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:811: done (4 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:812: done (3 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:813: done (2 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:814: done (1 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:815: done (0 left) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:816: read oid 46 snap -1 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:816: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:817: read oid 39 snap -1 2024-09-13T13:50:30.930 INFO:tasks.rados.rados.0.smithi159.stdout:817: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:818: read oid 23 snap -1 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:818: expect (ObjNum 91 snap 5 seq_num 91) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:819: rollback oid 13 current snap is 6 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 13 to 4 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:820: read oid 5 snap 3 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:820: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:821: read oid 35 snap -1 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:821: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:822: read oid 31 snap -1 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:822: expect (ObjNum 83 snap 4 seq_num 83) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:823: read oid 24 snap -1 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:823: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:30.931 INFO:tasks.rados.rados.0.smithi159.stdout:824: delete oid 6 current snap is 6 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:819: finishing rollback tid 0 to smithi15938319-13 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:819: finishing rollback tid 1 to smithi15938319-13 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:824: done (8 left) 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:825: tier_evict oid 3 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:825: tier_evict oid 3 snap -1 2024-09-13T13:50:30.938 INFO:tasks.rados.rados.0.smithi159.stdout:826: rollback oid 1 current snap is 6 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 1 to 4 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:827: read oid 12 snap -1 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:827: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:828: rollback oid 27 current snap is 6 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 27 to 5 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:819: finishing rollback tid 2 to smithi15938319-13 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 32 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:819: done (11 left) 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:829: read oid 29 snap -1 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:829: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:830: read oid 10 snap -1 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:830: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:30.939 INFO:tasks.rados.rados.0.smithi159.stdout:831: write oid 17 current snap is 6 2024-09-13T13:50:30.940 INFO:tasks.rados.rados.0.smithi159.stdout:831: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-13T13:50:30.943 INFO:tasks.rados.rados.0.smithi159.stdout:831: writing smithi15938319-17 from 463893 to 1026133 tid 1 2024-09-13T13:50:30.950 INFO:tasks.rados.rados.0.smithi159.stdout:831: writing smithi15938319-17 from 1593482 to 2251509 tid 2 2024-09-13T13:50:30.953 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:30.954 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:30.955 INFO:tasks.rados.rados.0.smithi159.stdout:831: writing smithi15938319-17 from 2685935 to 3245276 tid 3 2024-09-13T13:50:30.955 INFO:tasks.rados.rados.0.smithi159.stdout:832: write (excl) oid 20 current snap is 6 2024-09-13T13:50:30.955 INFO:tasks.rados.rados.0.smithi159.stdout:832: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-13T13:50:30.961 INFO:tasks.rados.rados.0.smithi159.stdout:832: writing smithi15938319-20 from 480700 to 1118189 tid 1 2024-09-13T13:50:30.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:30.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: writing smithi15938319-20 from 1651964 to 2053649 tid 2 2024-09-13T13:50:30.975 INFO:tasks.rados.rados.0.smithi159.stdout:832: writing smithi15938319-20 from 2713358 to 3483971 tid 3 2024-09-13T13:50:30.975 INFO:tasks.rados.rados.0.smithi159.stdout:833: read oid 49 snap -1 2024-09-13T13:50:30.975 INFO:tasks.rados.rados.0.smithi159.stdout:833: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:30.975 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:30.975 INFO:tasks.rados.rados.0.smithi159.stdout:816: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:31.006 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.023 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:31.022+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:31.036 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.065 INFO:tasks.rados.rados.0.smithi159.stdout:818: expect (ObjNum 91 snap 5 seq_num 91) 2024-09-13T13:50:31.065 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.112 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.112 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:31.123 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:31.122+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:31.217 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3244495 2024-09-13T13:50:31.224 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:31.224+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:826: finishing rollback tid 0 to smithi15938319-1 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:826: finishing rollback tid 1 to smithi15938319-1 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:826: finishing rollback tid 2 to smithi15938319-1 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 65 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:825: got (22) Invalid argument 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:828: finishing rollback tid 1 to smithi15938319-27 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 33 snap 0 seq_num 1540325968) dirty dne 2024-09-13T13:50:31.227 INFO:tasks.rados.rados.0.smithi159.stdout:820: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T13:50:31.229 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1287015 2024-09-13T13:50:31.231 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.231 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:31.243 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.267 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1308686 2024-09-13T13:50:31.280 INFO:tasks.rados.rados.0.smithi159.stdout:823: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:31.283 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 546217 2024-09-13T13:50:31.325 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:31.325+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:31.347 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.347 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:31.409 INFO:tasks.rados.rados.0.smithi159.stdout:827: expect (ObjNum 75 snap 3 seq_num 75) 2024-09-13T13:50:31.409 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.426 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:31.426+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:31.477 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2572287 2024-09-13T13:50:31.490 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.504 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.504 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.512 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:31.527 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:31.527+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:31.538 INFO:tasks.rados.rados.0.smithi159.stdout:821: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:31.541 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1517646 2024-09-13T13:50:31.612 INFO:tasks.rados.rados.0.smithi159.stdout:817: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:31.612 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.627 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.627 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.628 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:31.627+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:31.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:31.661 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.690 INFO:tasks.rados.rados.0.smithi159.stdout:832: finishing write tid 1 to smithi15938319-20 2024-09-13T13:50:31.691 INFO:tasks.rados.rados.0.smithi159.stdout:832: finishing write tid 2 to smithi15938319-20 2024-09-13T13:50:31.691 INFO:tasks.rados.rados.0.smithi159.stdout:822: expect (ObjNum 83 snap 4 seq_num 83) 2024-09-13T13:50:31.722 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:31.721+0000 7fc16c184640 -1 osd.11 55 handle_osd_map injecting map crc failure 2024-09-13T13:50:31.728 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:31.728+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:31.739 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.739 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:31.828 INFO:tasks.rados.rados.0.smithi159.stdout:829: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:31.828 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.829 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:31.829+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:31.850 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:31.851 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:31.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:31.883 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 987407 2024-09-13T13:50:31.918 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:31.930 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:31.930+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:31.966 INFO:tasks.rados.rados.0.smithi159.stdout:832: finishing write tid 3 to smithi15938319-20 2024-09-13T13:50:31.966 INFO:tasks.rados.rados.0.smithi159.stdout:832: finishing write tid 4 to smithi15938319-20 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: finishing write tid 5 to smithi15938319-20 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: oid 20 updating version 0 to 43 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: oid 20 version 43 is already newer than 41 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: oid 20 version 43 is already newer than 40 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: oid 20 version 43 is already newer than 42 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 43 (ObjNum 100 snap 6 seq_num 100) dirty exists 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:832: left oid 20 (ObjNum 100 snap 6 seq_num 100) 2024-09-13T13:50:31.967 INFO:tasks.rados.rados.0.smithi159.stdout:830: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:32.003 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.003 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.011 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:32.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:32.031 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:32.031+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: finishing write tid 1 to smithi15938319-17 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: finishing write tid 2 to smithi15938319-17 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: finishing write tid 3 to smithi15938319-17 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: finishing write tid 4 to smithi15938319-17 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: finishing write tid 5 to smithi15938319-17 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: oid 17 updating version 0 to 66 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: oid 17 version 66 is already newer than 65 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: oid 17 updating version 66 to 68 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: oid 17 version 68 is already newer than 67 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 68 (ObjNum 99 snap 6 seq_num 99) dirty exists 2024-09-13T13:50:32.034 INFO:tasks.rados.rados.0.smithi159.stdout:831: left oid 17 (ObjNum 99 snap 6 seq_num 99) 2024-09-13T13:50:32.035 INFO:tasks.rados.rados.0.smithi159.stdout:833: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T13:50:32.064 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1905780 2024-09-13T13:50:32.073 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:32.105 INFO:tasks.rados.rados.0.smithi159.stdout:816: done (15 left) 2024-09-13T13:50:32.105 INFO:tasks.rados.rados.0.smithi159.stdout:817: done (14 left) 2024-09-13T13:50:32.105 INFO:tasks.rados.rados.0.smithi159.stdout:818: done (13 left) 2024-09-13T13:50:32.105 INFO:tasks.rados.rados.0.smithi159.stdout:820: done (12 left) 2024-09-13T13:50:32.105 INFO:tasks.rados.rados.0.smithi159.stdout:821: done (11 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:822: done (10 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:823: done (9 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:825: done (8 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:826: done (7 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:827: done (6 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:828: done (5 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:829: done (4 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:830: done (3 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:831: done (2 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:832: done (1 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:833: done (0 left) 2024-09-13T13:50:32.106 INFO:tasks.rados.rados.0.smithi159.stdout:834: write oid 33 current snap is 6 2024-09-13T13:50:32.107 INFO:tasks.rados.rados.0.smithi159.stdout:834: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-13T13:50:32.110 INFO:tasks.rados.rados.0.smithi159.stdout:834: writing smithi15938319-33 from 497507 to 1210245 tid 1 2024-09-13T13:50:32.113 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.113 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.117 INFO:tasks.rados.rados.0.smithi159.stdout:834: writing smithi15938319-33 from 1710446 to 2372142 tid 2 2024-09-13T13:50:32.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:32.125 INFO:tasks.rados.rados.0.smithi159.stdout:834: writing smithi15938319-33 from 3140781 to 3697507 tid 3 2024-09-13T13:50:32.125 INFO:tasks.rados.rados.0.smithi159.stdout:835: read oid 37 snap -1 2024-09-13T13:50:32.125 INFO:tasks.rados.rados.0.smithi159.stdout:835: expect deleted 2024-09-13T13:50:32.125 INFO:tasks.rados.rados.0.smithi159.stdout:836: read oid 9 snap -1 2024-09-13T13:50:32.125 INFO:tasks.rados.rados.0.smithi159.stdout:836: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:837: read oid 18 snap -1 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:837: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:838: read oid 39 snap 3 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:838: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:839: read oid 35 snap -1 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:839: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:840: delete oid 26 current snap is 6 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:834: finishing write tid 1 to smithi15938319-33 2024-09-13T13:50:32.126 INFO:tasks.rados.rados.0.smithi159.stdout:834: finishing write tid 2 to smithi15938319-33 2024-09-13T13:50:32.127 INFO:tasks.rados.rados.0.smithi159.stdout:838: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-13T13:50:32.133 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:32.132+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:834: finishing write tid 3 to smithi15938319-33 2024-09-13T13:50:32.208 INFO:tasks.rados.rados.0.smithi159.stdout:836: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:32.210 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1671676 2024-09-13T13:50:32.224 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.224 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:32.234 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:32.232+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:32.245 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1163124 2024-09-13T13:50:32.256 INFO:tasks.rados.rados.0.smithi159.stdout:834: finishing write tid 4 to smithi15938319-33 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: finishing write tid 5 to smithi15938319-33 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: oid 33 updating version 0 to 49 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: oid 33 version 49 is already newer than 47 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: oid 33 version 49 is already newer than 48 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: oid 33 updating version 49 to 50 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 50 (ObjNum 101 snap 6 seq_num 101) dirty exists 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:834: left oid 33 (ObjNum 101 snap 6 seq_num 101) 2024-09-13T13:50:32.257 INFO:tasks.rados.rados.0.smithi159.stdout:837: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:32.333 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:32.332+0000 7f9b0c553640 -1 osd.12 55 handle_osd_map injecting map crc failure 2024-09-13T13:50:32.333 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:32.333+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:32.337 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.337 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:32.393 INFO:tasks.rados.rados.0.smithi159.stdout:839: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:32.393 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:32.435 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:32.434+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:32.450 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.450 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:834: done (6 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:835: done (5 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:836: done (4 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:837: done (3 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:838: done (2 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:839: done (1 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:840: done (0 left) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:841: read oid 14 snap -1 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:841: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:842: read oid 41 snap -1 2024-09-13T13:50:32.466 INFO:tasks.rados.rados.0.smithi159.stdout:842: expect deleted 2024-09-13T13:50:32.467 INFO:tasks.rados.rados.0.smithi159.stdout:843: read oid 8 snap -1 2024-09-13T13:50:32.467 INFO:tasks.rados.rados.0.smithi159.stdout:843: expect (ObjNum 94 snap 6 seq_num 94) 2024-09-13T13:50:32.467 INFO:tasks.rados.rados.0.smithi159.stdout:844: write (excl) oid 12 current snap is 6 2024-09-13T13:50:32.467 INFO:tasks.rados.rados.0.smithi159.stdout:844: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-13T13:50:32.471 INFO:tasks.rados.rados.0.smithi159.stdout:844: writing smithi15938319-12 from 531121 to 994357 tid 1 2024-09-13T13:50:32.478 INFO:tasks.rados.rados.0.smithi159.stdout:844: writing smithi15938319-12 from 1711057 to 2376422 tid 2 2024-09-13T13:50:32.486 INFO:tasks.rados.rados.0.smithi159.stdout:844: writing smithi15938319-12 from 3079274 to 3731121 tid 3 2024-09-13T13:50:32.486 INFO:tasks.rados.rados.0.smithi159.stdout:845: write (excl) oid 48 current snap is 6 2024-09-13T13:50:32.486 INFO:tasks.rados.rados.0.smithi159.stdout:845: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-13T13:50:32.491 INFO:tasks.rados.rados.0.smithi159.stdout:845: writing smithi15938319-48 from 547928 to 1086413 tid 1 2024-09-13T13:50:32.497 INFO:tasks.rados.rados.0.smithi159.stdout:845: writing smithi15938319-48 from 1769539 to 2178562 tid 2 2024-09-13T13:50:32.503 INFO:tasks.rados.rados.0.smithi159.stdout:845: writing smithi15938319-48 from 2706697 to 3238751 tid 3 2024-09-13T13:50:32.503 INFO:tasks.rados.rados.0.smithi159.stdout:846: read oid 13 snap -1 2024-09-13T13:50:32.503 INFO:tasks.rados.rados.0.smithi159.stdout:846: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:32.503 INFO:tasks.rados.rados.0.smithi159.stdout:847: read oid 35 snap 3 2024-09-13T13:50:32.503 INFO:tasks.rados.rados.0.smithi159.stdout:847: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:32.504 INFO:tasks.rados.rados.0.smithi159.stdout:848: delete oid 1 current snap is 6 2024-09-13T13:50:32.504 INFO:tasks.rados.rados.0.smithi159.stdout:841: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:32.504 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:32.536 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:32.536+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:32.562 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.563 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:32.583 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2955241 2024-09-13T13:50:32.618 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2854224 2024-09-13T13:50:32.629 INFO:tasks.rados.rados.0.smithi159.stdout:844: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:32.629 INFO:tasks.rados.rados.0.smithi159.stdout:843: expect (ObjNum 94 snap 6 seq_num 94) 2024-09-13T13:50:32.637 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:32.637+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:32.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:32.691 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 796759 2024-09-13T13:50:32.714 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.714 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.723 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:32.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:50:32.737 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:32.737+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:32.792 INFO:tasks.rados.rados.0.smithi159.stdout:844: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:32.792 INFO:tasks.rados.rados.0.smithi159.stdout:844: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:32.792 INFO:tasks.rados.rados.0.smithi159.stdout:844: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: finishing write tid 5 to smithi15938319-12 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: oid 12 updating version 0 to 48 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: oid 12 updating version 48 to 50 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: oid 12 version 50 is already newer than 49 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: oid 12 updating version 50 to 51 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 51 (ObjNum 103 snap 6 seq_num 103) dirty exists 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: left oid 12 (ObjNum 103 snap 6 seq_num 103) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:845: finishing write tid 1 to smithi15938319-48 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:841: done (7 left) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:842: done (6 left) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:843: done (5 left) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:844: done (4 left) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:848: done (3 left) 2024-09-13T13:50:32.793 INFO:tasks.rados.rados.0.smithi159.stdout:849: read oid 26 snap 5 2024-09-13T13:50:32.794 INFO:tasks.rados.rados.0.smithi159.stdout:849: expect (ObjNum 88 snap 5 seq_num 88) 2024-09-13T13:50:32.794 INFO:tasks.rados.rados.0.smithi159.stdout:850: tier_flush oid 7 2024-09-13T13:50:32.794 INFO:tasks.rados.rados.0.smithi159.stdout:850: tier_flush oid 7 snap -1 2024-09-13T13:50:32.794 INFO:tasks.rados.rados.0.smithi159.stdout:845: finishing write tid 2 to smithi15938319-48 2024-09-13T13:50:32.794 INFO:tasks.rados.rados.0.smithi159.stdout:846: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:32.795 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1269957 2024-09-13T13:50:32.818 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:32.818 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:32.826 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.827 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:32.837 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 620084 2024-09-13T13:50:32.838 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:32.837+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:32.840 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:32.840 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:32.852 INFO:tasks.rados.rados.0.smithi159.stdout:847: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:32.865 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:32.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:32.887 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:32.887 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:32.902 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:32.909 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:32.909 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:32.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: finishing write tid 3 to smithi15938319-48 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: finishing write tid 4 to smithi15938319-48 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: finishing write tid 5 to smithi15938319-48 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: oid 48 updating version 0 to 73 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: oid 48 version 73 is already newer than 71 2024-09-13T13:50:32.932 INFO:tasks.rados.rados.0.smithi159.stdout:845: oid 48 version 73 is already newer than 72 2024-09-13T13:50:32.933 INFO:tasks.rados.rados.0.smithi159.stdout:845: oid 48 version 73 is already newer than 70 2024-09-13T13:50:32.933 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 73 (ObjNum 104 snap 6 seq_num 104) dirty exists 2024-09-13T13:50:32.933 INFO:tasks.rados.rados.0.smithi159.stdout:845: left oid 48 (ObjNum 104 snap 6 seq_num 104) 2024-09-13T13:50:32.933 INFO:tasks.rados.rados.0.smithi159.stdout:845: done (4 left) 2024-09-13T13:50:32.933 INFO:tasks.rados.rados.0.smithi159.stdout:846: done (3 left) 2024-09-13T13:50:32.934 INFO:tasks.rados.rados.0.smithi159.stdout:847: done (2 left) 2024-09-13T13:50:32.934 INFO:tasks.rados.rados.0.smithi159.stdout:851: read oid 37 snap 4 2024-09-13T13:50:32.934 INFO:tasks.rados.rados.0.smithi159.stdout:851: expect deleted 2024-09-13T13:50:32.934 INFO:tasks.rados.rados.0.smithi159.stdout:852: set_chunk oid 44 target oid 2024-09-13T13:50:32.939 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:32.938+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:32.939 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:32.939 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:32.941 INFO:tasks.rados.rados.0.smithi159.stdout:852: set_chunk oid 44 offset: 1284096 length: 1315328 target oid 89e93071c15f29352375693664ba4e763e7bef9bf66ec7880742abc35b7b7c10 offset: 0 snap -1 2024-09-13T13:50:32.941 INFO:tasks.rados.rados.0.smithi159.stdout:853: set_chunk oid 28 target oid 2024-09-13T13:50:32.944 INFO:tasks.rados.rados.0.smithi159.stdout:853: set_chunk oid 28 offset: 139776 length: 250880 target oid 4bbfc0ad216260900011fd978d173dafe263b7079873e2f69f00ee77a50c1ce3 offset: 0 snap -1 2024-09-13T13:50:32.944 INFO:tasks.rados.rados.0.smithi159.stdout:854: write (excl) oid 34 current snap is 6 2024-09-13T13:50:32.944 INFO:tasks.rados.rados.0.smithi159.stdout:854: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-13T13:50:32.948 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:32.951 INFO:tasks.rados.rados.0.smithi159.stdout:854: writing smithi15938319-34 from 581542 to 1270525 tid 1 2024-09-13T13:50:32.952 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:32.952 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:32.956 INFO:tasks.rados.rados.0.smithi159.stdout:854: writing smithi15938319-34 from 1886503 to 2299195 tid 2 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:854: writing smithi15938319-34 from 2761543 to 3432494 tid 3 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:855: read oid 31 snap 3 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:855: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:856: tier_flush oid 10 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:856: tier_flush oid 10 snap -1 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:850: got (0) Success 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 0 (ObjNum 80 snap 3 seq_num 80) dirty exists 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:852: finishing set_chunk to oid 44 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 25 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-09-13T13:50:32.964 INFO:tasks.rados.rados.0.smithi159.stdout:853: finishing set_chunk to oid 28 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:854: finishing write tid 1 to smithi15938319-34 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:854: finishing write tid 2 to smithi15938319-34 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:850: done (7 left) 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:851: done (6 left) 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:852: done (5 left) 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:853: done (4 left) 2024-09-13T13:50:32.965 INFO:tasks.rados.rados.0.smithi159.stdout:857: read oid 15 snap -1 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:857: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:858: read oid 27 snap -1 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:858: expect deleted 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:859: tier_evict oid 42 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:859: tier_evict oid 42 snap -1 2024-09-13T13:50:32.966 INFO:tasks.rados.rados.0.smithi159.stdout:860: rollback oid 30 current snap is 6 2024-09-13T13:50:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 30 to 3 2024-09-13T13:50:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:861: snap_create 2024-09-13T13:50:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:859: got (22) Invalid argument 2024-09-13T13:50:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:860: finishing rollback tid 1 to smithi15938319-30 2024-09-13T13:50:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 0 (ObjNum 81 snap 0 seq_num 1061510647) dirty dne 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: finishing write tid 3 to smithi15938319-34 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: finishing write tid 4 to smithi15938319-34 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: finishing write tid 5 to smithi15938319-34 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: oid 34 updating version 0 to 32 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: oid 34 updating version 32 to 34 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: oid 34 version 34 is already newer than 33 2024-09-13T13:50:32.969 INFO:tasks.rados.rados.0.smithi159.stdout:854: oid 34 updating version 34 to 35 2024-09-13T13:50:32.970 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 35 (ObjNum 106 snap 6 seq_num 106) dirty exists 2024-09-13T13:50:32.970 INFO:tasks.rados.rados.0.smithi159.stdout:854: left oid 34 (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:32.973 INFO:tasks.rados.rados.0.smithi159.stdout:855: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-13T13:50:32.976 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:32.977 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:32.995 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.039 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:33.039+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:33.050 INFO:tasks.rados.rados.0.smithi159.stdout:857: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T13:50:33.050 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.055 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.055 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:33.055 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:33.055 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:33.075 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.125 INFO:tasks.rados.rados.0.smithi159.stdout:856: got (0) Success 2024-09-13T13:50:33.125 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T13:50:33.141 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:33.140+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:33.197 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:33.203 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:854: done (8 left) 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:855: done (7 left) 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:856: done (6 left) 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:857: done (5 left) 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:858: done (4 left) 2024-09-13T13:50:33.207 INFO:tasks.rados.rados.0.smithi159.stdout:859: done (3 left) 2024-09-13T13:50:33.208 INFO:tasks.rados.rados.0.smithi159.stdout:860: done (2 left) 2024-09-13T13:50:33.208 INFO:tasks.rados.rados.0.smithi159.stdout:861: done (1 left) 2024-09-13T13:50:33.208 INFO:tasks.rados.rados.0.smithi159.stdout:862: snap_create 2024-09-13T13:50:33.208 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.209 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:33.209 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:33.209 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:33.209 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:33.220 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:33.220 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:33.241 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:33.242 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:33.242 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:33.241+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:33.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:33.283 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:33.302 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 6] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:33.302 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:33.302 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T13:50:33.302 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T13:50:33.302 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T13:50:33.303 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-13T13:50:33.304 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:33.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:33.304+0000 7f7f327fc640 0 -- 172.21.15.27:0/4281799753 >> v2:172.21.15.122:6808/1255553857 conn(0x7f7f3421fa50 msgr2=0x7f7f34221e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.323 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:190 4.1 4:8bb695d2:::smithi15938319-48:head [write 2050536~739393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.705523+0000", 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.615909237, 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0055404599999999997, 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 190 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.705523+0000", 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.705523+0000", 2024-09-13T13:50:33.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.705524+0000", 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706553+0000", 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010288350000000001 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706562+0000", 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3209999999999994e-06 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706569+0000", 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6259999999999997e-06 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706646+0000", 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7018e-05 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706692+0000", 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6372000000000003e-05 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.706766+0000", 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3955000000000006e-05 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.708279+0000", 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001512869 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.710992+0000", 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027129720000000001 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.711023+0000", 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1334000000000002e-05 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.711064+0000", 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0512000000000001e-05 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.10.22:7 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.572247367999999, 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072188460000000001, 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4400000000000001e-07 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749188+0000", 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.644e-06 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749258+0000", 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9744000000000002e-05 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749263+0000", 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6339999999999998e-06 2024-09-13T13:50:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752867+0000", 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036041469999999998 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752952+0000", 2024-09-13T13:50:33.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4035000000000004e-05 2024-09-13T13:50:33.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756404+0000", 2024-09-13T13:50:33.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034522979999999999 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.423266851000001, 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.336 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:50:33.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:50:33.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:50:33.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:265 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.409987544, 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082114400000000004, 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 265 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911447+0000", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911457+0000", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0433e-05 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911476+0000", 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8612000000000001e-05 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911486+0000", 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911546+0000", 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9877000000000003e-05 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911627+0000", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1546999999999997e-05 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917585+0000", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059573229999999996 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918841+0000", 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001255969 2024-09-13T13:50:33.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919557+0000", 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000716079 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919613+0000", 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5881000000000003e-05 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919656+0000", 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3643999999999998e-05 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.407639236000001, 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.350 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:50:33.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:50:33.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:33.356 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.398927561000001, 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:50:33.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:50:33.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.551381597999999, 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:50:33.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:50:33.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:50:33.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:50:33.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:334 4.1 4:80c58c2d:::smithi15938319-35:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.773761+0000", 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.547670970999999, 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058139589999999996, 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 334 2024-09-13T13:50:33.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773761+0000", 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773761+0000", 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773762+0000", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5799999999999998e-07 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773769+0000", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0609999999999998e-06 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773780+0000", 2024-09-13T13:50:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0560999999999999e-05 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773788+0000", 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9009999999999994e-06 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773851+0000", 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3596000000000006e-05 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.773929+0000", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7340000000000002e-05 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778456+0000", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045276509999999997 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778890+0000", 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00043333199999999998 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.779522+0000", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063280300000000003 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.779551+0000", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8839999999999998e-05 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.779575+0000", 2024-09-13T13:50:33.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4015999999999998e-05 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:366 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.246511858000002, 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079177420000000002, 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 366 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074921+0000", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074925+0000", 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8010000000000001e-06 2024-09-13T13:50:33.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1880000000000002e-06 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074939+0000", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075137+0000", 2024-09-13T13:50:33.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197927 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075206+0000", 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8806999999999994e-05 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.078002+0000", 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002796078 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.081901+0000", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003899713 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082743+0000", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084112100000000001 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082793+0000", 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0359999999999999e-05 2024-09-13T13:50:33.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082838+0000", 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5321000000000002e-05 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:378 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.245469840999998, 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071838889999999997, 2024-09-13T13:50:33.382 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 378 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075964+0000", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0160000000000001e-06 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075970+0000", 2024-09-13T13:50:33.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9090000000000003e-06 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075981+0000", 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1042e-05 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075990+0000", 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2539999999999992e-06 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076042+0000", 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1999000000000001e-05 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076116+0000", 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4060000000000006e-05 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.079664+0000", 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035479420000000001 2024-09-13T13:50:33.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082307+0000", 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002643472 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083060+0000", 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075247599999999997 2024-09-13T13:50:33.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083109+0000", 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9387e-05 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083146+0000", 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7332000000000001e-05 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.388 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1038 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 9.5414085649999993, 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078733310000000008, 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.389 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1038 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780025+0000", 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.238e-06 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780029+0000", 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1470000000000003e-06 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780040+0000", 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0597e-05 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780049+0000", 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3260000000000002e-06 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780132+0000", 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3185999999999999e-05 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780189+0000", 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7234000000000001e-05 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.787897+0000", 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077076030000000004 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1246 4.1 4:81ab2613:::smithi15938319-29:head [write 2873212~588997] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0698709930000003, 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067622710000000003, 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1246 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251563+0000", 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.767e-06 2024-09-13T13:50:33.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252186+0000", 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062314599999999999 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252194+0000", 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1929999999999993e-06 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252200+0000", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0839999999999999e-06 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252247+0000", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7852000000000002e-05 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252301+0000", 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4085000000000001e-05 2024-09-13T13:50:33.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252403+0000", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101209 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258157+0000", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057543159999999998 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258206+0000", 2024-09-13T13:50:33.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8809e-05 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258285+0000", 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8881999999999995e-05 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258324+0000", 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8927999999999997e-05 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1247 4.1 4:81ab2613:::smithi15938319-29:head [setxattr _header (56) in=7b,truncate 3462209] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0692214079999998, 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006237584, 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1247 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252212+0000", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.06e-07 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252214+0000", 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8480000000000001e-06 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252223+0000", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4069999999999993e-06 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252227+0000", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.109e-06 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252920+0000", 2024-09-13T13:50:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00069335899999999995 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252981+0000", 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1035000000000001e-05 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253083+0000", 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101807 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258162+0000", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00507874 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258375+0000", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021354999999999999 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258410+0000", 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4857999999999997e-05 2024-09-13T13:50:33.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258449+0000", 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8265e-05 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1248 4.1 4:81ab2613:::smithi15938319-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0691924430000004, 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063470940000000002, 2024-09-13T13:50:33.406 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1248 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0999999999999999e-07 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252243+0000", 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2500000000000001e-06 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252248+0000", 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.609e-06 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252254+0000", 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4330000000000003e-06 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253282+0000", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010284160000000001 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253338+0000", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6440000000000002e-05 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258482+0000", 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005143997 2024-09-13T13:50:33.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258532+0000", 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9527999999999997e-05 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258587+0000", 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5010999999999999e-05 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1250 4.1 4:80c58c2d:::smithi15938319-35:head [read 0~2571438,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252273+0000", 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0691594609999999, 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005889694, 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.412 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1250 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252273+0000", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252273+0000", 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252273+0000", 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5600000000000002e-07 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252275+0000", 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.68e-06 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252278+0000", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5470000000000001e-06 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252281+0000", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5890000000000001e-06 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.257098+0000", 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048175359999999999 2024-09-13T13:50:33.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.257201+0000", 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102692 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258163+0000", 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00096139399999999996 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1251 4.1 4:80c58c2d:::smithi15938319-35:head [read 649379~1922059,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0691480889999996, 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067450039999999998, 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1251 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.417 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-07 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252286+0000", 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.452e-06 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252289+0000", 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.196e-06 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252292+0000", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6129999999999998e-06 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258607+0000", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063152319999999996 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258663+0000", 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5918999999999998e-05 2024-09-13T13:50:33.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.259029+0000", 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000366375 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.574641411, 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.421 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.425 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing(4.f epoch 53/34 rep_tid 118 entries 53'57 (0'0) error 4:f883ee18:::smithi15938319-40:head by client.4571.0:1290 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 51'54)", 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.5718054050000001, 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084851660000000006, 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749628+0000", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0400000000000005e-07 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749637+0000", 2024-09-13T13:50:33.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749672+0000", 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4409999999999998e-05 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749685+0000", 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2754e-05 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754728+0000", 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050430300000000004 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758112+0000", 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033844600000000002 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:33.429 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 3.3599711320000001, 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:50:33.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:50:33.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.35816950600000003, 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:50:33.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.439 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.441 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:33.441 INFO:teuthology.orchestra.run.smithi159.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T13:50:33.441 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:33.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:33.443 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:33.440+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:33.449 INFO:tasks.rados.rados.0.smithi159.stdout:849: expect (ObjNum 88 snap 5 seq_num 88) 2024-09-13T13:50:33.527 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:33.527 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:33.541 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:33.540+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:33.544 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.544 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:33.545 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:33.545 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:33.553 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:33.586 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2126566 2024-09-13T13:50:33.642 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:33.642+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:33.696 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:33.696 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:33.719 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:33.718+0000 7fe0fbe07640 -1 osd.2 57 handle_osd_map injecting map crc failure 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:849: done (1 left) 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:862: done (0 left) 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:863: set_chunk oid 41 target oid 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:863: set_chunk oid 41 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:864: read oid 28 snap 7 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:864: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T13:50:33.721 INFO:tasks.rados.rados.0.smithi159.stdout:865: read oid 47 snap -1 2024-09-13T13:50:33.722 INFO:tasks.rados.rados.0.smithi159.stdout:865: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:33.722 INFO:tasks.rados.rados.0.smithi159.stdout:866: write oid 31 current snap is 8 2024-09-13T13:50:33.722 INFO:tasks.rados.rados.0.smithi159.stdout:866: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-13T13:50:33.727 INFO:tasks.rados.rados.0.smithi159.stdout:866: writing smithi15938319-31 from 598349 to 1078934 tid 1 2024-09-13T13:50:33.728 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:33.728+0000 7f1a526ee640 -1 osd.13 56 handle_osd_map injecting map crc failure 2024-09-13T13:50:33.733 INFO:tasks.rados.rados.0.smithi159.stdout:866: writing smithi15938319-31 from 1544985 to 2217688 tid 2 2024-09-13T13:50:33.740 INFO:tasks.rados.rados.0.smithi159.stdout:866: writing smithi15938319-31 from 2905319 to 3387542 tid 3 2024-09-13T13:50:33.740 INFO:tasks.rados.rados.0.smithi159.stdout:867: read oid 18 snap -1 2024-09-13T13:50:33.740 INFO:tasks.rados.rados.0.smithi159.stdout:867: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:33.740 INFO:tasks.rados.rados.0.smithi159.stdout:868: tier_evict oid 1 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:868: tier_evict oid 1 snap -1 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:863: finishing set_chunk to oid 41 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:863: got expected ENOENT (src dne) 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:863: done (5 left) 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:869: delete oid 8 current snap is 8 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:865: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T13:50:33.741 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.743 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:33.742+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:33.802 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 158937 2024-09-13T13:50:33.813 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 45.589088222000001, 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31765422599999998, 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.814 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222553+0000", 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222555+0000", 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3489999999999999e-06 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222562+0000", 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222572+0000", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.031e-05 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222600+0000", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7647000000000001e-05 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222658+0000", 2024-09-13T13:50:33.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8520000000000002e-05 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.222671+0000", 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2588e-05 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538469+0000", 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.31579794900000002 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538626+0000", 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000157496 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.538781+0000", 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015506799999999999 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.539522+0000", 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074055599999999996 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540140+0000", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061783300000000003 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540177+0000", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7011999999999997e-05 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.540208+0000", 2024-09-13T13:50:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0735999999999998e-05 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 45.269971697999999, 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.821 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:50:33.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.215645809000002, 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:50:33.827 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:50:33.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:50:33.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:50:33.833 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.202267419000002, 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.837 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:270 4.4 4:21c00763:::smithi15938319-21:head [assert-version v6,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.895827713999999, 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20992622599999999, 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 270 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915814+0000", 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915816+0000", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.88e-06 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915822+0000", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8769999999999998e-06 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915839+0000", 2024-09-13T13:50:33.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7663999999999999e-05 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915855+0000", 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5627000000000001e-05 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915941+0000", 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5984999999999998e-05 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.916024+0000", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3361000000000004e-05 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.124703+0000", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.208678681 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.125590+0000", 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088729200000000003 2024-09-13T13:50:33.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.125656+0000", 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6229000000000006e-05 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.125698+0000", 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1919000000000003e-05 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.125740+0000", 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1711000000000001e-05 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.388934482, 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:50:33.844 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.845 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.845 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.849 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5835224749999997, 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:50:33.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:50:33.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5834853029999998, 2024-09-13T13:50:33.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:50:33.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:50:33.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5834645749999998, 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:50:33.861 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:50:33.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:50:33.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5833682739999997, 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.867 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:33.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:50:33.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5833456300000002, 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.873 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:50:33.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5833196000000003, 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.879 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:50:33.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:50:33.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:33.885 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4438187329999996, 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:50:33.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.891 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4437117859999997, 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:50:33.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.443682527, 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.898 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:33.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:50:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:50:33.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4373181239999999, 2024-09-13T13:50:33.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:33.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:50:33.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:50:33.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4323579300000002, 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:50:33.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:50:33.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:50:33.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1279 4.4 4:22f57a3c:::smithi15938319-42:head [write 2555481~425187] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4275137979999997, 2024-09-13T13:50:33.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.37714365399999999, 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1279 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:33.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384129+0000", 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384523+0000", 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039437499999999998 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384535+0000", 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1626e-05 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384541+0000", 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4699999999999999e-06 2024-09-13T13:50:33.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1349e-05 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384589+0000", 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3249999999999996e-06 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728981+0000", 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34439230599999998 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729015+0000", 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4149000000000001e-05 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740625+0000", 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011609895 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740676+0000", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.13e-05 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754660+0000", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013983423 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754774+0000", 2024-09-13T13:50:33.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114554 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754954+0000", 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017924600000000001 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759913+0000", 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049594189999999996 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761144+0000", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001231174 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761182+0000", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8235999999999998e-05 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761271+0000", 2024-09-13T13:50:33.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8882999999999996e-05 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1280 4.4 4:22f57a3c:::smithi15938319-42:head [setxattr _header (56) in=7b,truncate 3512630] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4270868009999997, 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379144494, 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1280 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:33.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.99e-07 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384559+0000", 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384564+0000", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0610000000000003e-06 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384569+0000", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384617+0000", 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8479000000000001e-05 2024-09-13T13:50:33.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384623+0000", 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.677e-06 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729055+0000", 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34443185999999998 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729104+0000", 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9335000000000001e-05 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740720+0000", 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011616191 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740766+0000", 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5667000000000002e-05 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759391+0000", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018624900999999999 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759450+0000", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8891000000000002e-05 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759563+0000", 2024-09-13T13:50:33.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011324600000000001 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761528+0000", 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019654220000000001 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763564+0000", 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020357410000000002 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:33.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763611+0000", 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6873000000000003e-05 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763699+0000", 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8288000000000006e-05 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1281 4.4 4:22f57a3c:::smithi15938319-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:33.938 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4270594230000002, 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379374818, 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1281 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:33.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1899999999999998e-07 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384585+0000", 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.334e-06 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384591+0000", 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8259999999999998e-06 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384596+0000", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6439999999999997e-06 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384640+0000", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3217999999999999e-05 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384643+0000", 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6330000000000002e-06 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729130+0000", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34448669500000001 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729160+0000", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0035999999999998e-05 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740807+0000", 2024-09-13T13:50:33.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011646844 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740902+0000", 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4952999999999997e-05 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760235+0000", 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019333605 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:33.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760386+0000", 2024-09-13T13:50:33.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015056100000000001 2024-09-13T13:50:33.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763739+0000", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033526620000000002 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763822+0000", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3255000000000002e-05 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763957+0000", 2024-09-13T13:50:33.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013513300000000001 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:33.947 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:33.948 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:33.948+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:33.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:865: done (5 left) 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:869: done (4 left) 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:870: read oid 17 snap 5 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:870: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:871: read oid 34 snap -1 2024-09-13T13:50:33.950 INFO:tasks.rados.rados.0.smithi159.stdout:871: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:872: tier_flush oid 12 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:872: tier_flush oid 12 snap -1 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:864: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:868: got (22) Invalid argument 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:866: finishing write tid 1 to smithi15938319-31 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:866: finishing write tid 2 to smithi15938319-31 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:864: done (6 left) 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:868: done (5 left) 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:873: set_chunk oid 16 target oid 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:873: set_chunk oid 16 offset: 772608 length: 741888 target oid db3ad804eeb197cac43b7e3adeb4e551be51bca4943ef4a3fc1b121972c84378 offset: 0 snap -1 2024-09-13T13:50:33.951 INFO:tasks.rados.rados.0.smithi159.stdout:874: snap_remove snap 3 2024-09-13T13:50:34.049 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:34.048+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:34.094 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.094 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:34.094 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:34.094 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:34.150 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:34.150+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:34.205 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.206 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:34.206 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:34.206 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:34.206 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:34.251 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:34.251+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:34.318 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.318 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.579619775000001, 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:50:34.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:50:34.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574539248000001, 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024188794, 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.325 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742198+0000", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742205+0000", 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:50:34.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742227+0000", 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2688999999999999e-05 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742237+0000", 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0151e-05 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742312+0000", 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4913999999999995e-05 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for blocked object", 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742364+0000", 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1774999999999998e-05 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756421+0000", 2024-09-13T13:50:34.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014057106999999999 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756508+0000", 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6533999999999995e-05 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766385+0000", 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098774239999999992 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.331 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574379152999999, 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:50:34.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:50:34.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:50:34.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.339 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574327838999999, 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:50:34.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:50:34.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574255642000001, 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.346 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:50:34.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:50:34.352 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574216636999999, 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.353 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:50:34.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.359 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:213 3.3 3:d7665af5:::smithi15938319-14 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574189126, 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024017457999999998, 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.360 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 213 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742547+0000", 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6800000000000001e-07 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742550+0000", 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.965e-06 2024-09-13T13:50:34.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742570+0000", 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0207000000000002e-05 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742576+0000", 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8889999999999999e-06 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743625+0000", 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001049407 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743730+0000", 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105025 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.751509+0000", 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077789039999999997 2024-09-13T13:50:34.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766480+0000", 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014970343000000001 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766513+0000", 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3009000000000002e-05 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766540+0000", 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766564+0000", 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3799e-05 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.366 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.574152224999999, 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.367 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:34.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:50:34.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:201 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.560816701, 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010542467, 2024-09-13T13:50:34.373 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755920+0000", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.513e-06 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755926+0000", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5509999999999999e-06 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755953+0000", 2024-09-13T13:50:34.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6687999999999999e-05 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755972+0000", 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9550000000000001e-05 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766321+0000", 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010349205 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766329+0000", 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3479999999999998e-06 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766461+0000", 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132612 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.377 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.548802128999998, 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.378 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:50:34.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:203 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.542294650999999, 2024-09-13T13:50:34.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019371698999999999, 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:34.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5199999999999998e-07 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774444+0000", 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5600000000000001e-06 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793662+0000", 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019218452 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793669+0000", 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793716+0000", 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6953999999999998e-05 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793722+0000", 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793813+0000", 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0705999999999995e-05 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:221 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.523095615999999, 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018779990999999999, 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793641+0000", 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.316e-06 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793646+0000", 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7870000000000001e-06 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812292+0000", 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018646276 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812298+0000", 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.152e-06 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812336+0000", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8336e-05 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812344+0000", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3909999999999998e-06 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812420+0000", 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5733000000000001e-05 2024-09-13T13:50:34.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:234 3.3 3:cdbb9bf3:::smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.518397422, 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021808134999999999, 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 234 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798339+0000", 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1990000000000001e-06 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798344+0000", 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798354+0000", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8670000000000006e-06 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798359+0000", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0309999999999998e-06 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798399+0000", 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0191999999999997e-05 2024-09-13T13:50:34.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798514+0000", 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114484 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.803569+0000", 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005055492 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808174+0000", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046051089999999996 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820010+0000", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011835215 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820079+0000", 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9377999999999999e-05 2024-09-13T13:50:34.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820146+0000", 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7336999999999996e-05 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.518241305, 2024-09-13T13:50:34.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.496914717999999, 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.406 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:50:34.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.418698441, 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:50:34.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:50:34.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:50:34.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:250 3.3 3:edb81fa6:::smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.418628336000001, 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019084286999999998, 2024-09-13T13:50:34.417 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 250 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898108+0000", 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6699999999999999e-07 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898111+0000", 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8710000000000001e-06 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898131+0000", 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9939e-05 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898136+0000", 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8599999999999998e-06 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898476+0000", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033951499999999998 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898641+0000", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016495999999999999 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909561+0000", 2024-09-13T13:50:34.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010919988 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.916236+0000", 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066746660000000001 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917085+0000", 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084985200000000001 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917156+0000", 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0246999999999993e-05 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917192+0000", 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5921999999999999e-05 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.6.0:178 4.5 e45/34 4:a1652551:::smithi15938319-33:head v 45'26, mlcod=42'24)", 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.667241753999999, 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085953929999999998, 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:34.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649495+0000", 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5600000000000004e-07 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651055+0000", 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015604760000000001 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651121+0000", 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5664e-05 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651135+0000", 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4263e-05 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651207+0000", 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2053999999999998e-05 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651244+0000", 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7113999999999997e-05 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.658045+0000", 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0068001379999999998 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.658089+0000", 2024-09-13T13:50:34.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4728000000000003e-05 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1149 4.5 4:a1652551:::smithi15938319-33:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 9.3325807510000001, 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016672093999999998, 2024-09-13T13:50:34.429 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1149 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:34.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984157+0000", 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2819999999999999e-06 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984165+0000", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7500000000000003e-06 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984182+0000", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7207e-05 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984192+0000", 2024-09-13T13:50:34.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0237999999999999e-05 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984249+0000", 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6409999999999997e-05 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984311+0000", 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2706999999999994e-05 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:25.000827+0000", 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016515499999999999 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1530 3.3 3:d56e57c3:::db3ad804eeb197cac43b7e3adeb4e551be51bca4943ef4a3fc1b121972c84378:head [write 0~741888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:33.924648+0000", 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.39208714500000003, 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081195120000000006, 2024-09-13T13:50:34.434 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1530 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.924648+0000", 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.924648+0000", 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.924650+0000", 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.792e-06 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.925448+0000", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079741899999999995 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.925461+0000", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3417999999999999e-05 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.925468+0000", 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6800000000000004e-06 2024-09-13T13:50:34.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.925542+0000", 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3998999999999994e-05 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.925546+0000", 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.929809+0000", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042631179999999998 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.929890+0000", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1596999999999998e-05 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.929972+0000", 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1699000000000005e-05 2024-09-13T13:50:34.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.932066+0000", 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020942309999999998 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.932710+0000", 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064409199999999997 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.932743+0000", 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2407999999999999e-05 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.932768+0000", 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5148999999999999e-05 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.441 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:34.444 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:34.442+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:34.543 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:34.543+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:34.587 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.587 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:34.587 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:34.587 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:34.644 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:34.644+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:34.699 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.699 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:34.699 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:34.699 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:34.699 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:34.727 INFO:tasks.rados.rados.0.smithi159.stdout:874: done (6 left) 2024-09-13T13:50:34.727 INFO:tasks.rados.rados.0.smithi159.stdout:875: read oid 35 snap 7 2024-09-13T13:50:34.727 INFO:tasks.rados.rados.0.smithi159.stdout:875: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:876: read oid 24 snap 5 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:876: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:877: rollback oid 3 current snap is 8 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 3 to 7 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: finishing write tid 3 to smithi15938319-31 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: finishing write tid 4 to smithi15938319-31 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: finishing write tid 5 to smithi15938319-31 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: oid 31 updating version 0 to 61 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: oid 31 updating version 61 to 62 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: oid 31 version 62 is already newer than 59 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: oid 31 version 62 is already newer than 60 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 62 (ObjNum 107 snap 8 seq_num 107) dirty exists 2024-09-13T13:50:34.728 INFO:tasks.rados.rados.0.smithi159.stdout:866: left oid 31 (ObjNum 107 snap 8 seq_num 107) 2024-09-13T13:50:34.729 INFO:tasks.rados.rados.0.smithi159.stdout:867: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:34.729 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:34.729 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:34.726+0000 7faeaa210640 -1 osd.8 58 handle_osd_map injecting map crc failure 2024-09-13T13:50:34.745 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:34.745+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:34.786 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1246121 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:34.817 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 44.587177381000004, 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:50:34.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:50:34.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:50:34.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.824 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.082439199, 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.825 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:50:34.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:50:34.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:34.831 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.082389887000001, 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:50:34.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:50:34.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.837 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.082374182999999, 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.838 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:50:34.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.844 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.082346507, 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:50:34.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:50:34.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:216 3.2 3:683f8235:::smithi15938319-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.050275763999998, 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025464730000000001, 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.851 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 216 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764531+0000", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764533+0000", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0590000000000001e-06 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764542+0000", 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4010000000000003e-06 2024-09-13T13:50:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764562+0000", 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9720000000000001e-05 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764571+0000", 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0110000000000006e-06 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764631+0000", 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0647999999999999e-05 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764752+0000", 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012058 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774799+0000", 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010046602 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.777512+0000", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027135890000000002 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789910+0000", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012397905000000001 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789952+0000", 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1604999999999999e-05 2024-09-13T13:50:34.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.789995+0000", 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3609999999999998e-05 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.047128909000001, 2024-09-13T13:50:34.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:34.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:50:34.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:50:34.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.040059927000001, 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.864 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:34.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:50:34.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:50:34.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.005521465000001, 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.871 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:50:34.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:50:34.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:50:34.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.997891623000001, 2024-09-13T13:50:34.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:50:34.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:50:34.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.883 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.994515250999999, 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.889 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:247 3.2 3:501d6036:::smithi15938319-48 low_tier:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.916772652999999, 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013184006999999999, 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 247 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898034+0000", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898036+0000", 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5189999999999999e-06 2024-09-13T13:50:34.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898049+0000", 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.275e-05 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898073+0000", 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4029999999999999e-05 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898086+0000", 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.256e-05 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898164+0000", 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8712000000000001e-05 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898311+0000", 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014640599999999999 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.901880+0000", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035688629999999998 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.907582+0000", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057028859999999999 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911105+0000", 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035227000000000001 2024-09-13T13:50:34.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0890999999999997e-05 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911218+0000", 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1690000000000001e-05 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:249 3.2 3:5e5aa6e9:::smithi15938319-50 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.916386092, 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026560466000000001, 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 249 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.896 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898420+0000", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898421+0000", 2024-09-13T13:50:34.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898426+0000", 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7740000000000004e-06 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898437+0000", 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0868e-05 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898443+0000", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5409999999999997e-06 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898782+0000", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033879599999999999 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898917+0000", 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013452200000000001 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915774+0000", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016857014999999999 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919261+0000", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034877839999999998 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924925+0000", 2024-09-13T13:50:34.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0056640450000000004 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924955+0000", 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9561999999999999e-05 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.924981+0000", 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5860999999999999e-05 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:252 4.b 4:def5de2b:::smithi15938319-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.916337721000001, 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017224822000000001, 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 252 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.902 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898469+0000", 2024-09-13T13:50:34.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.06e-07 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898475+0000", 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0560000000000001e-06 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898505+0000", 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9603000000000001e-05 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898511+0000", 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1850000000000001e-06 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898567+0000", 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5523999999999999e-05 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898640+0000", 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3752999999999997e-05 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.912256+0000", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01361593 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.914820+0000", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025639759999999999 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915602+0000", 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078219100000000001 2024-09-13T13:50:34.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915651+0000", 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8911e-05 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915694+0000", 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2187000000000001e-05 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:253 4.b 4:d4fa90c1:::smithi15938319-4:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.916265129999999, 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022974749999999999, 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 253 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.908 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898541+0000", 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898542+0000", 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6000000000000003e-07 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898552+0000", 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0005e-05 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898562+0000", 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6819999999999998e-06 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898568+0000", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4400000000000002e-06 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899164+0000", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059608899999999995 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899245+0000", 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0457999999999994e-05 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918743+0000", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019497970999999999 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921290+0000", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025477830000000001 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921459+0000", 2024-09-13T13:50:34.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016888999999999999 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921490+0000", 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0629000000000001e-05 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.921516+0000", 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6143e-05 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:256 4.b 4:d7a30b61:::smithi15938319-7:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.916224266, 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016599936999999999, 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 256 2024-09-13T13:50:34.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898582+0000", 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898583+0000", 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1600000000000001e-07 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898586+0000", 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1480000000000002e-06 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898592+0000", 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2790000000000004e-06 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898598+0000", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899379+0000", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078147899999999996 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.899420+0000", 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1026999999999999e-05 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911795+0000", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012374444 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.912502+0000", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070700199999999998 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915080+0000", 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002577986 2024-09-13T13:50:34.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915136+0000", 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6214000000000001e-05 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.915182+0000", 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5957000000000001e-05 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.891655970999999, 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:50:34.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:50:34.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:50:34.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.680632430999999, 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:34.926 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:34.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.669117148000002, 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.6.22:33 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:19.173205+0000", 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.641601802, 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011030195, 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173205+0000", 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173205+0000", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173208+0000", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173716+0000", 2024-09-13T13:50:34.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050801799999999995 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173800+0000", 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3502000000000001e-05 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173813+0000", 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3244000000000001e-05 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173985+0000", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000171958 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.173993+0000", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.178971+0000", 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049780409999999999 2024-09-13T13:50:34.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.179087+0000", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116588 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.179342+0000", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00025456200000000001 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.181562+0000", 2024-09-13T13:50:34.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022196849999999999 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.184130+0000", 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025684839999999998 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.184190+0000", 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9825999999999999e-05 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:34.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.184235+0000", 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4826000000000001e-05 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:34.941 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:866: done (8 left) 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:867: done (7 left) 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:878: read oid 6 snap -1 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:878: expect deleted 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:872: got (16) Device or resource busy 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 12 v 51 (ObjNum 103 snap 6 seq_num 103) unflushed 6 2024-09-13T13:50:34.943 INFO:tasks.rados.rados.0.smithi159.stdout:870: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-13T13:50:34.944 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 573766 2024-09-13T13:50:34.944 INFO:tasks.rados.rados.0.smithi159.stdout:871: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:34.944 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:34.944 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:34.942+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:34.984 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 537955 2024-09-13T13:50:35.043 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:35.043+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:35.049 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:35.049 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:35.049 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:35.049 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:35.058 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:35.081 INFO:tasks.rados.rados.0.smithi159.stdout:873: finishing set_chunk to oid 16 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 772608 length: 741888 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:877: finishing rollback tid 0 to smithi15938319-3 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:877: finishing rollback tid 1 to smithi15938319-3 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:879: read oid 8 snap -1 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:879: expect deleted 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:870: done (8 left) 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:871: done (7 left) 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:872: done (6 left) 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:873: done (5 left) 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:880: read oid 18 snap 4 2024-09-13T13:50:35.082 INFO:tasks.rados.rados.0.smithi159.stdout:880: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:35.083 INFO:tasks.rados.rados.0.smithi159.stdout:881: set_chunk oid 49 target oid 2024-09-13T13:50:35.101 INFO:tasks.rados.rados.0.smithi159.stdout:881: set_chunk oid 49 offset: 603136 length: 1948672 target oid 8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338 offset: 0 snap -1 2024-09-13T13:50:35.102 INFO:tasks.rados.rados.0.smithi159.stdout:882: delete oid 23 current snap is 8 2024-09-13T13:50:35.102 INFO:tasks.rados.rados.0.smithi159.stdout:877: finishing rollback tid 2 to smithi15938319-3 2024-09-13T13:50:35.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 62 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:35.102 INFO:tasks.rados.rados.0.smithi159.stdout:875: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:35.102 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:35.131 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:35.144 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:35.143+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:35.175 INFO:tasks.rados.rados.0.smithi159.stdout:876: expect (ObjNum 56 snap 3 seq_num 56) 2024-09-13T13:50:35.187 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:35.188 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:35.188 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:35.188 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:35.188 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:35.196 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:35.218 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:50:35.262 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:35.262+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:35.300 INFO:tasks.rados.rados.0.smithi159.stdout:875: done (7 left) 2024-09-13T13:50:35.300 INFO:tasks.rados.rados.0.smithi159.stdout:876: done (6 left) 2024-09-13T13:50:35.301 INFO:tasks.rados.rados.0.smithi159.stdout:877: done (5 left) 2024-09-13T13:50:35.301 INFO:tasks.rados.rados.0.smithi159.stdout:882: done (4 left) 2024-09-13T13:50:35.301 INFO:tasks.rados.rados.0.smithi159.stdout:883: write (excl) oid 21 current snap is 8 2024-09-13T13:50:35.301 INFO:tasks.rados.rados.0.smithi159.stdout:883: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-13T13:50:35.307 INFO:tasks.rados.rados.0.smithi159.stdout:883: writing smithi15938319-21 from 648770 to 1355102 tid 1 2024-09-13T13:50:35.314 INFO:tasks.rados.rados.0.smithi159.stdout:883: writing smithi15938319-21 from 2120431 to 2540461 tid 2 2024-09-13T13:50:35.321 INFO:tasks.rados.rados.0.smithi159.stdout:883: writing smithi15938319-21 from 3271235 to 3703627 tid 3 2024-09-13T13:50:35.321 INFO:tasks.rados.rados.0.smithi159.stdout:878: done (4 left) 2024-09-13T13:50:35.321 INFO:tasks.rados.rados.0.smithi159.stdout:884: read oid 30 snap 6 2024-09-13T13:50:35.321 INFO:tasks.rados.rados.0.smithi159.stdout:884: expect deleted 2024-09-13T13:50:35.321 INFO:tasks.rados.rados.0.smithi159.stdout:885: read oid 40 snap -1 2024-09-13T13:50:35.322 INFO:tasks.rados.rados.0.smithi159.stdout:885: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:35.322 INFO:tasks.rados.rados.0.smithi159.stdout:886: rollback oid 48 current snap is 8 2024-09-13T13:50:35.322 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 48 to 7 2024-09-13T13:50:35.322 INFO:tasks.rados.rados.0.smithi159.stdout:880: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:35.322 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:35.343 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.609370274, 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.344 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:50:35.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.609292873000001, 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.351 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.352 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:50:35.353 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.354 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.355 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:50:35.356 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.609140334999999, 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.357 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:35.358 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.359 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.360 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:50:35.361 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.362 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.609034296000001, 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.363 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:50:35.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:50:35.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:35.369 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.600695217999998, 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.370 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:50:35.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:207 3.2 e41/33 3:755d2207:::smithi15938319-8 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.59593997, 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022139559, 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.374 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745615+0000", 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745616+0000", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5199999999999996e-07 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751147+0000", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055306139999999997 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751393+0000", 2024-09-13T13:50:35.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024623899999999998 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751399+0000", 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6490000000000002e-06 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751448+0000", 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9016e-05 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751472+0000", 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3175000000000001e-05 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767716+0000", 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016244341999999998 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767755+0000", 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8772000000000001e-05 2024-09-13T13:50:35.378 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.573328507999999, 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.379 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:35.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:50:35.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:50:35.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:35.385 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.57156191, 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:35.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:50:35.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.391 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.566682568000001, 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:50:35.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:50:35.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:50:35.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.397 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:219 3.2 e41/33 3:650cb820:::smithi15938319-20 low_tier:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.559591584, 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02650433, 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:35.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781965+0000", 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.452e-06 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789891+0000", 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079261269999999998 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789958+0000", 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6587999999999993e-05 2024-09-13T13:50:35.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789965+0000", 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790010+0000", 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4592000000000003e-05 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790030+0000", 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0630000000000001e-05 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808419+0000", 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018388315999999998 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808468+0000", 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9168000000000001e-05 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.402 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.556170843, 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.403 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:50:35.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:50:35.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:227 3.2 e41/33 3:626302dd:::smithi15938319-28 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.551659356999998, 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024242393000000001, 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.409 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789897+0000", 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8999999999999995e-07 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795632+0000", 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057353760000000004 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795680+0000", 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8420000000000001e-05 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795692+0000", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1452e-05 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795737+0000", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5596000000000002e-05 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795751+0000", 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3127e-05 2024-09-13T13:50:35.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814104+0000", 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018353103999999999 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814138+0000", 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4527999999999999e-05 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:233 3.0 3:3563ec55:::smithi15938319-34 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.545628614000002, 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020873143, 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 233 2024-09-13T13:50:35.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795927+0000", 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795928+0000", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899999999999999e-06 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795935+0000", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795945+0000", 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0378e-05 2024-09-13T13:50:35.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795953+0000", 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8509999999999999e-06 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798390+0000", 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024377449999999998 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798511+0000", 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000120852 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.805311+0000", 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067998520000000003 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816555+0000", 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011243510999999999 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816742+0000", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00018714599999999999 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816766+0000", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4450000000000001e-05 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816800+0000", 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3405000000000003e-05 2024-09-13T13:50:35.419 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.531913552999999, 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.420 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:50:35.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:50:35.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.529028508, 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.426 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:50:35.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:50:35.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.526824482999999, 2024-09-13T13:50:35.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:50:35.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:50:35.438 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.519833839, 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:50:35.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:50:35.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:245 3.2 e41/33 3:60c9a0cb:::smithi15938319-46 low_tier:head v 41'9, mlcod=41'6)", 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.514146852, 2024-09-13T13:50:35.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024215621999999999, 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827409+0000", 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832412+0000", 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050024800000000001 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832526+0000", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113959 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832535+0000", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0850000000000003e-06 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832620+0000", 2024-09-13T13:50:35.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5050999999999996e-05 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832644+0000", 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4012e-05 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851590+0000", 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018946647 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851624+0000", 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3628999999999999e-05 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:243 3.2 e41/33 3:6893a2ac:::smithi15938319-44 low_tier:head v 41'10, mlcod=41'6)", 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.509137891000002, 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019400412999999998, 2024-09-13T13:50:35.448 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832417+0000", 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832418+0000", 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837463+0000", 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050444909999999999 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837489+0000", 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6551e-05 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.837498+0000", 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5920000000000001e-06 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839367+0000", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0018689220000000001 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839391+0000", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4811e-05 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851794+0000", 2024-09-13T13:50:35.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012402827999999999 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851818+0000", 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3459999999999999e-05 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:35.453 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 22.414842225000001, 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:35.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:35.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:50:35.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:35.459 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:35.461 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:35.462 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:35.462 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1158216 2024-09-13T13:50:35.462 INFO:tasks.rados.rados.0.smithi159.stdout:881: finishing set_chunk to oid 49 2024-09-13T13:50:35.462 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 603136 length: 1948672 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:879: done (6 left) 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:880: done (5 left) 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:881: done (4 left) 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:887: read oid 50 snap -1 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:887: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:888: read oid 19 snap -1 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:888: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:889: read oid 37 snap -1 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:889: expect deleted 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:890: read oid 8 snap -1 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:890: expect deleted 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:891: read oid 18 snap -1 2024-09-13T13:50:35.463 INFO:tasks.rados.rados.0.smithi159.stdout:891: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:35.464 INFO:tasks.rados.rados.0.smithi159.stdout:892: write (excl) oid 2 current snap is 8 2024-09-13T13:50:35.464 INFO:tasks.rados.rados.0.smithi159.stdout:883: finishing write tid 1 to smithi15938319-21 2024-09-13T13:50:35.464 INFO:tasks.rados.rados.0.smithi159.stdout:892: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-13T13:50:35.464 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:35.460+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:35.466 INFO:tasks.rados.rados.0.smithi159.stdout:892: writing smithi15938319-2 from 665577 to 1447158 tid 1 2024-09-13T13:50:35.473 INFO:tasks.rados.rados.0.smithi159.stdout:892: writing smithi15938319-2 from 2062560 to 2626248 tid 2 2024-09-13T13:50:35.480 INFO:tasks.rados.rados.0.smithi159.stdout:892: writing smithi15938319-2 from 3182305 to 3825969 tid 3 2024-09-13T13:50:35.480 INFO:tasks.rados.rados.0.smithi159.stdout:893: write oid 39 current snap is 8 2024-09-13T13:50:35.480 INFO:tasks.rados.rados.0.smithi159.stdout:893: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-13T13:50:35.486 INFO:tasks.rados.rados.0.smithi159.stdout:893: writing smithi15938319-39 from 682384 to 1139214 tid 1 2024-09-13T13:50:35.491 INFO:tasks.rados.rados.0.smithi159.stdout:893: writing smithi15938319-39 from 1721042 to 2144741 tid 2 2024-09-13T13:50:35.497 INFO:tasks.rados.rados.0.smithi159.stdout:893: writing smithi15938319-39 from 2809728 to 3264664 tid 3 2024-09-13T13:50:35.501 INFO:tasks.rados.rados.0.smithi159.stdout:893: writing smithi15938319-39 from 3731357 to 3882384 tid 4 2024-09-13T13:50:35.501 INFO:tasks.rados.rados.0.smithi159.stdout:894: set_chunk oid 36 target oid 2024-09-13T13:50:35.509 INFO:tasks.rados.rados.0.smithi159.stdout:894: set_chunk oid 36 offset: 905216 length: 482304 target oid f3b6345ca2d8865c085e64c13a1c649a3b709064ce2215787042154144611c20 offset: 0 snap -1 2024-09-13T13:50:35.509 INFO:tasks.rados.rados.0.smithi159.stdout:895: write oid 24 current snap is 8 2024-09-13T13:50:35.509 INFO:tasks.rados.rados.0.smithi159.stdout:895: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-13T13:50:35.515 INFO:tasks.rados.rados.0.smithi159.stdout:895: writing smithi15938319-24 from 699191 to 1231270 tid 1 2024-09-13T13:50:35.521 INFO:tasks.rados.rados.0.smithi159.stdout:895: writing smithi15938319-24 from 1779524 to 2346881 tid 2 2024-09-13T13:50:35.528 INFO:tasks.rados.rados.0.smithi159.stdout:895: writing smithi15938319-24 from 2837151 to 3503359 tid 3 2024-09-13T13:50:35.528 INFO:tasks.rados.rados.0.smithi159.stdout:896: read oid 12 snap 7 2024-09-13T13:50:35.528 INFO:tasks.rados.rados.0.smithi159.stdout:896: expect (ObjNum 103 snap 6 seq_num 103) 2024-09-13T13:50:35.528 INFO:tasks.rados.rados.0.smithi159.stdout:897: write (excl) oid 15 current snap is 8 2024-09-13T13:50:35.528 INFO:tasks.rados.rados.0.smithi159.stdout:883: finishing write tid 2 to smithi15938319-21 2024-09-13T13:50:35.529 INFO:tasks.rados.rados.0.smithi159.stdout:883: finishing write tid 3 to smithi15938319-21 2024-09-13T13:50:35.529 INFO:tasks.rados.rados.0.smithi159.stdout:897: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-13T13:50:35.535 INFO:tasks.rados.rados.0.smithi159.stdout:897: writing smithi15938319-15 from 715998 to 1323326 tid 1 2024-09-13T13:50:35.540 INFO:tasks.rados.rados.0.smithi159.stdout:897: writing smithi15938319-15 from 1838006 to 2265374 tid 2 2024-09-13T13:50:35.546 INFO:tasks.rados.rados.0.smithi159.stdout:897: writing smithi15938319-15 from 2864574 to 3342054 tid 3 2024-09-13T13:50:35.550 INFO:tasks.rados.rados.0.smithi159.stdout:897: writing smithi15938319-15 from 3863835 to 3915998 tid 4 2024-09-13T13:50:35.550 INFO:tasks.rados.rados.0.smithi159.stdout:884: done (14 left) 2024-09-13T13:50:35.550 INFO:tasks.rados.rados.0.smithi159.stdout:898: read oid 32 snap -1 2024-09-13T13:50:35.551 INFO:tasks.rados.rados.0.smithi159.stdout:898: expect (ObjNum 93 snap 5 seq_num 93) 2024-09-13T13:50:35.551 INFO:tasks.rados.rados.0.smithi159.stdout:899: read oid 42 snap -1 2024-09-13T13:50:35.551 INFO:tasks.rados.rados.0.smithi159.stdout:899: expect (ObjNum 90 snap 5 seq_num 90) 2024-09-13T13:50:35.551 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:50:35.552 INFO:tasks.rados.rados.0.smithi159.stdout:886: finishing rollback tid 0 to smithi15938319-48 2024-09-13T13:50:35.552 INFO:tasks.rados.rados.0.smithi159.stdout:886: finishing rollback tid 1 to smithi15938319-48 2024-09-13T13:50:35.552 INFO:tasks.rados.rados.0.smithi159.stdout:886: finishing rollback tid 2 to smithi15938319-48 2024-09-13T13:50:35.552 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 84 (ObjNum 104 snap 6 seq_num 104) dirty exists 2024-09-13T13:50:35.552 INFO:tasks.rados.rados.0.smithi159.stdout:885: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:35.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:35.561 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:35.560+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:35.586 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:35.586 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:35.586 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:35.586 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:35.597 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:35.613 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 735901 2024-09-13T13:50:35.634 INFO:tasks.rados.rados.0.smithi159.stdout:885: done (15 left) 2024-09-13T13:50:35.635 INFO:tasks.rados.rados.0.smithi159.stdout:886: done (14 left) 2024-09-13T13:50:35.635 INFO:tasks.rados.rados.0.smithi159.stdout:889: done (13 left) 2024-09-13T13:50:35.635 INFO:tasks.rados.rados.0.smithi159.stdout:890: done (12 left) 2024-09-13T13:50:35.635 INFO:tasks.rados.rados.0.smithi159.stdout:900: snap_remove snap 4 2024-09-13T13:50:35.661 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:35.660+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:35.696 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:35.697 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:35.717 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:35.717 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:35.717 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:35.717 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:35.717 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:35.719 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:35.720 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:35.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:35.741 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:35.762 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:35.763 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:35.763 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:35.762+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:35.783 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:35.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:35.805 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:35.826 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:35.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:35.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:35.864 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:35.863+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:35.868 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:35.868 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:35.870 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:35.870 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:35.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:35.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:35.890 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:35.911 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:35.911 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:35.933 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:35.953 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:35.954 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:35.964 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:35.963+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:35.976 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:35.976+0000 7fd3d492b640 0 -- 172.21.15.27:0/2978138239 >> v2:172.21.15.122:6808/1255553857 conn(0x7fd3d021f570 msgr2=0x7fd3d0221960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:35.981 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:35.981 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:35.990 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:36.065 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:36.064+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:36.092 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.092 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.101 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:36.167 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:36.166+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:36.199 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:36.199 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:36.204 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.204 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.212 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:900: done (12 left) 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:901: read oid 26 snap 5 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:901: expect (ObjNum 88 snap 5 seq_num 88) 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:902: read oid 8 snap -1 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:902: expect deleted 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:887: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-13T13:50:36.218 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:36.221 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:36.225 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:36.224+0000 7faeaa210640 -1 osd.8 59 handle_osd_map injecting map crc failure 2024-09-13T13:50:36.225 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:36.224+0000 7fc16a180640 -1 osd.11 59 handle_osd_map injecting map crc failure 2024-09-13T13:50:36.268 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:36.268+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:36.297 INFO:tasks.rados.rados.0.smithi159.stdout:891: expect (ObjNum 81 snap 4 seq_num 81) 2024-09-13T13:50:36.312 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1879892 2024-09-13T13:50:36.326 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.326 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.334 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:36.369 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:36.369+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:36.374 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:36.423 INFO:tasks.rados.rados.0.smithi159.stdout:892: finishing write tid 1 to smithi15938319-2 2024-09-13T13:50:36.424 INFO:tasks.rados.rados.0.smithi159.stdout:888: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:36.470 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:36.469+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:36.477 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.477 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:36.503 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2641037 2024-09-13T13:50:36.546 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2918560 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:892: finishing write tid 2 to smithi15938319-2 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:903: read oid 46 snap -1 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:903: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:887: done (14 left) 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:888: done (13 left) 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:891: done (12 left) 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:904: read oid 9 snap -1 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:904: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:36.565 INFO:tasks.rados.rados.0.smithi159.stdout:892: finishing write tid 3 to smithi15938319-2 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: finishing write tid 4 to smithi15938319-2 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: finishing write tid 5 to smithi15938319-2 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: oid 2 updating version 0 to 63 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: oid 2 version 63 is already newer than 62 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: oid 2 version 63 is already newer than 61 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: oid 2 updating version 63 to 64 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 64 (ObjNum 111 snap 8 seq_num 111) dirty exists 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: left oid 2 (ObjNum 111 snap 8 seq_num 111) 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 1 to smithi15938319-39 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:905: read oid 40 snap -1 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:905: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:892: done (13 left) 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:906: write oid 47 current snap is 8 2024-09-13T13:50:36.566 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 2 to smithi15938319-39 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 3 to smithi15938319-39 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 4 to smithi15938319-39 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 5 to smithi15938319-39 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: finishing write tid 6 to smithi15938319-39 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: oid 39 updating version 0 to 68 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: oid 39 updating version 68 to 70 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: oid 39 version 70 is already newer than 67 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: oid 39 version 70 is already newer than 69 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: oid 39 version 70 is already newer than 66 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 70 (ObjNum 112 snap 8 seq_num 112) dirty exists 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:893: left oid 39 (ObjNum 112 snap 8 seq_num 112) 2024-09-13T13:50:36.567 INFO:tasks.rados.rados.0.smithi159.stdout:894: finishing set_chunk to oid 36 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 905216 length: 482304 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:895: finishing write tid 1 to smithi15938319-24 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:895: finishing write tid 2 to smithi15938319-24 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:895: finishing write tid 3 to smithi15938319-24 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:895: finishing write tid 4 to smithi15938319-24 2024-09-13T13:50:36.568 INFO:tasks.rados.rados.0.smithi159.stdout:906: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-13T13:50:36.571 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:36.571+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:36.574 INFO:tasks.rados.rados.0.smithi159.stdout:906: writing smithi15938319-47 from 732805 to 1531735 tid 1 2024-09-13T13:50:36.582 INFO:tasks.rados.rados.0.smithi159.stdout:906: writing smithi15938319-47 from 2296488 to 2867514 tid 2 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:906: writing smithi15938319-47 from 3291997 to 3697102 tid 3 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:893: done (13 left) 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:894: done (12 left) 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:907: read oid 14 snap 5 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:907: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:908: read oid 1 snap -1 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:908: expect deleted 2024-09-13T13:50:36.588 INFO:tasks.rados.rados.0.smithi159.stdout:909: delete oid 16 current snap is 8 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: finishing write tid 5 to smithi15938319-24 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: oid 24 updating version 0 to 46 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: oid 24 version 46 is already newer than 45 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: oid 24 version 46 is already newer than 44 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: oid 24 version 46 is already newer than 43 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 46 (ObjNum 113 snap 8 seq_num 113) dirty exists 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:895: left oid 24 (ObjNum 113 snap 8 seq_num 113) 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:896: expect (ObjNum 103 snap 6 seq_num 103) 2024-09-13T13:50:36.589 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:36.620 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.620 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:36.672 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:36.672+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:36.718 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:36.738 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.738 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:36.773 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:36.772+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:36.785 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 1 to smithi15938319-15 2024-09-13T13:50:36.785 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 2 to smithi15938319-15 2024-09-13T13:50:36.785 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 3 to smithi15938319-15 2024-09-13T13:50:36.785 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 4 to smithi15938319-15 2024-09-13T13:50:36.785 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 5 to smithi15938319-15 2024-09-13T13:50:36.786 INFO:tasks.rados.rados.0.smithi159.stdout:898: expect (ObjNum 93 snap 5 seq_num 93) 2024-09-13T13:50:36.808 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2644762 2024-09-13T13:50:36.853 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.853 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:36.905 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:36.905+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: finishing write tid 6 to smithi15938319-15 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: oid 15 updating version 0 to 57 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: oid 15 updating version 57 to 60 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: oid 15 version 60 is already newer than 59 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: oid 15 version 60 is already newer than 58 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: oid 15 version 60 is already newer than 56 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 60 (ObjNum 114 snap 8 seq_num 114) dirty exists 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:897: left oid 15 (ObjNum 114 snap 8 seq_num 114) 2024-09-13T13:50:36.933 INFO:tasks.rados.rados.0.smithi159.stdout:899: expect (ObjNum 90 snap 5 seq_num 90) 2024-09-13T13:50:36.968 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:36.969 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:36.976 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:36.976+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:36.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:37.078 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:37.077+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:37.125 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.125 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.129 INFO:tasks.rados.rados.0.smithi159.stdout:883: finishing write tid 4 to smithi15938319-21 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: finishing write tid 5 to smithi15938319-21 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: oid 21 updating version 0 to 47 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: oid 21 updating version 47 to 48 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: oid 21 version 48 is already newer than 46 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: oid 21 version 48 is already newer than 45 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 48 (ObjNum 110 snap 8 seq_num 110) dirty exists 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: left oid 21 (ObjNum 110 snap 8 seq_num 110) 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:883: done (14 left) 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:895: done (13 left) 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:896: done (12 left) 2024-09-13T13:50:37.130 INFO:tasks.rados.rados.0.smithi159.stdout:897: done (11 left) 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:898: done (10 left) 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:899: done (9 left) 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:909: done (8 left) 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:910: write (excl) oid 5 current snap is 8 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:901: expect (ObjNum 88 snap 5 seq_num 88) 2024-09-13T13:50:37.131 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:37.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:37.178 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:37.177+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:37.194 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:37.279 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:37.279+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:37.282 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.282 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:37.326 INFO:tasks.rados.rados.0.smithi159.stdout:903: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T13:50:37.332 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1328878 2024-09-13T13:50:37.347 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:37.380 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:37.380+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:904: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:37.398 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.398 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:37.422 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 492751 2024-09-13T13:50:37.442 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 93429 2024-09-13T13:50:37.465 INFO:tasks.rados.rados.0.smithi159.stdout:905: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T13:50:37.481 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:37.481+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:37.483 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:37.512 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: finishing write tid 1 to smithi15938319-47 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: finishing write tid 2 to smithi15938319-47 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: finishing write tid 3 to smithi15938319-47 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: finishing write tid 4 to smithi15938319-47 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: finishing write tid 5 to smithi15938319-47 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: oid 47 updating version 0 to 43 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: oid 47 version 43 is already newer than 42 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: oid 47 version 43 is already newer than 41 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: oid 47 updating version 43 to 44 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 44 (ObjNum 115 snap 8 seq_num 115) dirty exists 2024-09-13T13:50:37.541 INFO:tasks.rados.rados.0.smithi159.stdout:906: left oid 47 (ObjNum 115 snap 8 seq_num 115) 2024-09-13T13:50:37.542 INFO:tasks.rados.rados.0.smithi159.stdout:907: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:37.554 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.555 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.556 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1933499 2024-09-13T13:50:37.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:37.606 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:37.606+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:37.670 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.670 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:37.679 INFO:tasks.rados.rados.0.smithi159.stdout:910: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-13T13:50:37.683 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:37.682+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:37.685 INFO:tasks.rados.rados.0.smithi159.stdout:910: writing smithi15938319-5 from 766419 to 1315847 tid 1 2024-09-13T13:50:37.692 INFO:tasks.rados.rados.0.smithi159.stdout:910: writing smithi15938319-5 from 2013452 to 2588147 tid 2 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:910: writing smithi15938319-5 from 3346843 to 3774492 tid 3 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:901: done (8 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:902: done (7 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:903: done (6 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:904: done (5 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:905: done (4 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:906: done (3 left) 2024-09-13T13:50:37.699 INFO:tasks.rados.rados.0.smithi159.stdout:907: done (2 left) 2024-09-13T13:50:37.700 INFO:tasks.rados.rados.0.smithi159.stdout:908: done (1 left) 2024-09-13T13:50:37.700 INFO:tasks.rados.rados.0.smithi159.stdout:911: delete oid 40 current snap is 8 2024-09-13T13:50:37.700 INFO:tasks.rados.rados.0.smithi159.stdout:910: finishing write tid 1 to smithi15938319-5 2024-09-13T13:50:37.700 INFO:tasks.rados.rados.0.smithi159.stdout:910: finishing write tid 2 to smithi15938319-5 2024-09-13T13:50:37.702 INFO:tasks.rados.rados.0.smithi159.stdout:910: finishing write tid 3 to smithi15938319-5 2024-09-13T13:50:37.702 INFO:tasks.rados.rados.0.smithi159.stdout:910: finishing write tid 4 to smithi15938319-5 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: finishing write tid 5 to smithi15938319-5 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: oid 5 updating version 0 to 46 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: oid 5 updating version 46 to 49 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: oid 5 version 49 is already newer than 47 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: oid 5 version 49 is already newer than 48 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 49 (ObjNum 117 snap 8 seq_num 117) dirty exists 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: left oid 5 (ObjNum 117 snap 8 seq_num 117) 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:910: done (1 left) 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:911: done (0 left) 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:912: write (excl) oid 41 current snap is 8 2024-09-13T13:50:37.703 INFO:tasks.rados.rados.0.smithi159.stdout:912: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-13T13:50:37.708 INFO:tasks.rados.rados.0.smithi159.stdout:912: writing smithi15938319-41 from 400033 to 1099959 tid 1 2024-09-13T13:50:37.713 INFO:tasks.rados.rados.0.smithi159.stdout:912: writing smithi15938319-41 from 1614063 to 2000033 tid 2 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:913: read oid 33 snap 5 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:913: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:914: tier_promote oid 6 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:912: finishing write tid 1 to smithi15938319-41 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:915: write oid 50 current snap is 8 2024-09-13T13:50:37.714 INFO:tasks.rados.rados.0.smithi159.stdout:915: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-13T13:50:37.719 INFO:tasks.rados.rados.0.smithi159.stdout:915: writing smithi15938319-50 from 416840 to 1192015 tid 1 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:915: writing smithi15938319-50 from 1672545 to 2016840 tid 2 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:916: set_chunk oid 19 target oid 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:914: got (0) Success 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 7 (ObjNum 3457 snap 0 seq_num 1540298096) clean dne 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: finishing write tid 2 to smithi15938319-41 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: finishing write tid 3 to smithi15938319-41 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: finishing write tid 4 to smithi15938319-41 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: oid 41 updating version 0 to 65 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: oid 41 updating version 65 to 67 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:912: oid 41 version 67 is already newer than 66 2024-09-13T13:50:37.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 67 (ObjNum 119 snap 8 seq_num 119) dirty exists 2024-09-13T13:50:37.726 INFO:tasks.rados.rados.0.smithi159.stdout:912: left oid 41 (ObjNum 119 snap 8 seq_num 119) 2024-09-13T13:50:37.728 INFO:tasks.rados.rados.0.smithi159.stdout:916: set_chunk oid 19 offset: 2316800 length: 184320 target oid 249bd22a0c56c061c80e98a24a9440f0b129b01bdf6c1be9c7e64230a5e43acf offset: 0 snap -1 2024-09-13T13:50:37.728 INFO:tasks.rados.rados.0.smithi159.stdout:912: done (4 left) 2024-09-13T13:50:37.728 INFO:tasks.rados.rados.0.smithi159.stdout:914: done (3 left) 2024-09-13T13:50:37.728 INFO:tasks.rados.rados.0.smithi159.stdout:917: write oid 6 current snap is 8 2024-09-13T13:50:37.728 INFO:tasks.rados.rados.0.smithi159.stdout:917: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-13T13:50:37.732 INFO:tasks.rados.rados.0.smithi159.stdout:917: writing smithi15938319-6 from 433647 to 884071 tid 1 2024-09-13T13:50:37.738 INFO:tasks.rados.rados.0.smithi159.stdout:917: writing smithi15938319-6 from 1331027 to 1913060 tid 2 2024-09-13T13:50:37.738 INFO:tasks.rados.rados.0.smithi159.stdout:918: write (excl) oid 39 current snap is 8 2024-09-13T13:50:37.738 INFO:tasks.rados.rados.0.smithi159.stdout:913: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:37.741 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 553711 2024-09-13T13:50:37.767 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 806805 2024-09-13T13:50:37.784 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:37.783+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:37.786 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 148256 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: finishing write tid 1 to smithi15938319-50 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: finishing write tid 2 to smithi15938319-50 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: finishing write tid 3 to smithi15938319-50 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: finishing write tid 4 to smithi15938319-50 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: oid 50 updating version 0 to 52 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: oid 50 version 52 is already newer than 51 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: oid 50 version 52 is already newer than 50 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 52 (ObjNum 120 snap 8 seq_num 120) dirty exists 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:915: left oid 50 (ObjNum 120 snap 8 seq_num 120) 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:916: finishing set_chunk to oid 19 2024-09-13T13:50:37.814 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 52 (ObjNum 89 snap 5 seq_num 89) dirty exists 2024-09-13T13:50:37.815 INFO:tasks.rados.rados.0.smithi159.stdout:917: finishing write tid 1 to smithi15938319-6 2024-09-13T13:50:37.815 INFO:tasks.rados.rados.0.smithi159.stdout:917: finishing write tid 2 to smithi15938319-6 2024-09-13T13:50:37.815 INFO:tasks.rados.rados.0.smithi159.stdout:918: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-13T13:50:37.819 INFO:tasks.rados.rados.0.smithi159.stdout:918: writing smithi15938319-39 from 450454 to 1092480 tid 1 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:918: writing smithi15938319-39 from 1505862 to 2050454 tid 2 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:913: done (4 left) 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:915: done (3 left) 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:916: done (2 left) 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:919: read oid 23 snap -1 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:919: expect deleted 2024-09-13T13:50:37.825 INFO:tasks.rados.rados.0.smithi159.stdout:920: set_chunk oid 42 target oid 2024-09-13T13:50:37.826 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.827 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.829 INFO:tasks.rados.rados.0.smithi159.stdout:920: set_chunk oid 42 offset: 1898496 length: 490496 target oid 7b68a1579d6a7ea3f90fd4821f42ecad39ec7d0e548d55c0566e85f13f95a192 offset: 0 snap 5 2024-09-13T13:50:37.829 INFO:tasks.rados.rados.0.smithi159.stdout:921: write oid 18 current snap is 8 2024-09-13T13:50:37.829 INFO:tasks.rados.rados.0.smithi159.stdout:917: finishing write tid 3 to smithi15938319-6 2024-09-13T13:50:37.830 INFO:tasks.rados.rados.0.smithi159.stdout:921: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-13T13:50:37.835 INFO:tasks.rados.rados.0.smithi159.stdout:921: writing smithi15938319-18 from 467261 to 1184536 tid 1 2024-09-13T13:50:37.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:37.840 INFO:tasks.rados.rados.0.smithi159.stdout:921: writing smithi15938319-18 from 1847991 to 2067261 tid 2 2024-09-13T13:50:37.841 INFO:tasks.rados.rados.0.smithi159.stdout:922: read oid 3 snap -1 2024-09-13T13:50:37.841 INFO:tasks.rados.rados.0.smithi159.stdout:922: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:37.841 INFO:tasks.rados.rados.0.smithi159.stdout:923: set_chunk oid 13 target oid 2024-09-13T13:50:37.848 INFO:tasks.rados.rados.0.smithi159.stdout:923: set_chunk oid 13 offset: 704000 length: 199680 target oid 278c2036742fb1092f9d1b560c7db926a4a7353d3d192e99f1e5a73c53250acc offset: 0 snap -1 2024-09-13T13:50:37.848 INFO:tasks.rados.rados.0.smithi159.stdout:924: snap_create 2024-09-13T13:50:37.848 INFO:tasks.rados.rados.0.smithi159.stdout:917: finishing write tid 4 to smithi15938319-6 2024-09-13T13:50:37.848 INFO:tasks.rados.rados.0.smithi159.stdout:917: oid 6 updating version 0 to 13 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:917: oid 6 updating version 13 to 15 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:917: oid 6 version 15 is already newer than 14 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 15 (ObjNum 121 snap 8 seq_num 121) dirty exists 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:917: left oid 6 (ObjNum 121 snap 8 seq_num 121) 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: finishing write tid 1 to smithi15938319-39 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: finishing write tid 2 to smithi15938319-39 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: finishing write tid 3 to smithi15938319-39 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: finishing write tid 4 to smithi15938319-39 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: oid 39 updating version 0 to 79 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: oid 39 version 79 is already newer than 78 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: oid 39 version 79 is already newer than 77 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 79 (ObjNum 122 snap 8 seq_num 122) dirty exists 2024-09-13T13:50:37.849 INFO:tasks.rados.rados.0.smithi159.stdout:918: left oid 39 (ObjNum 122 snap 8 seq_num 122) 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:920: finishing set_chunk to oid 42 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: finishing write tid 1 to smithi15938319-18 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: finishing write tid 2 to smithi15938319-18 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: finishing write tid 3 to smithi15938319-18 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: finishing write tid 4 to smithi15938319-18 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: oid 18 updating version 0 to 54 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: oid 18 updating version 54 to 56 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: oid 18 version 56 is already newer than 55 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 56 (ObjNum 123 snap 8 seq_num 123) dirty exists 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:921: left oid 18 (ObjNum 123 snap 8 seq_num 123) 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:922: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:37.850 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 301665 2024-09-13T13:50:37.870 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1318024 2024-09-13T13:50:37.885 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:37.884+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:37.899 INFO:tasks.rados.rados.0.smithi159.stdout:923: finishing set_chunk to oid 13 2024-09-13T13:50:37.900 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 704000 length: 199680 2024-09-13T13:50:37.900 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T13:50:37.944 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:37.944 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:37.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:37.986 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:37.985+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:38.061 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.061 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:38.087 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:38.086+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:38.175 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.175 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:38.188 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:38.187+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:38.224 INFO:tasks.rados.rados.0.smithi159.stdout:917: done (7 left) 2024-09-13T13:50:38.224 INFO:tasks.rados.rados.0.smithi159.stdout:918: done (6 left) 2024-09-13T13:50:38.224 INFO:tasks.rados.rados.0.smithi159.stdout:919: done (5 left) 2024-09-13T13:50:38.224 INFO:tasks.rados.rados.0.smithi159.stdout:920: done (4 left) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:921: done (3 left) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:922: done (2 left) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:923: done (1 left) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:924: done (0 left) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:925: tier_promote oid 5 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:926: read oid 19 snap -1 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:926: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:927: delete oid 39 current snap is 9 2024-09-13T13:50:38.225 INFO:tasks.rados.rados.0.smithi159.stdout:925: got (0) Success 2024-09-13T13:50:38.226 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 49 (ObjNum 117 snap 8 seq_num 117) dirty exists 2024-09-13T13:50:38.229 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:38.228+0000 7fe0f9e03640 -1 osd.2 60 handle_osd_map injecting map crc failure 2024-09-13T13:50:38.229 INFO:tasks.rados.rados.0.smithi159.stdout:926: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:38.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:50:38.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:38.288 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:38.287+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:38.299 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.300 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:925: done (2 left) 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:926: done (1 left) 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:927: done (0 left) 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:928: read oid 2 snap -1 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:928: expect (ObjNum 111 snap 8 seq_num 111) 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:929: read oid 36 snap -1 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:929: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:930: write (excl) oid 30 current snap is 9 2024-09-13T13:50:38.383 INFO:tasks.rados.rados.0.smithi159.stdout:930: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-13T13:50:38.388 INFO:tasks.rados.rados.0.smithi159.stdout:930: writing smithi15938319-30 from 500875 to 968648 tid 1 2024-09-13T13:50:38.389 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:38.388+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:38.395 INFO:tasks.rados.rados.0.smithi159.stdout:930: writing smithi15938319-30 from 1564955 to 2100875 tid 2 2024-09-13T13:50:38.395 INFO:tasks.rados.rados.0.smithi159.stdout:931: write (excl) oid 9 current snap is 9 2024-09-13T13:50:38.395 INFO:tasks.rados.rados.0.smithi159.stdout:931: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-13T13:50:38.401 INFO:tasks.rados.rados.0.smithi159.stdout:931: writing smithi15938319-9 from 517682 to 1060704 tid 1 2024-09-13T13:50:38.407 INFO:tasks.rados.rados.0.smithi159.stdout:931: writing smithi15938319-9 from 1623437 to 2117682 tid 2 2024-09-13T13:50:38.407 INFO:tasks.rados.rados.0.smithi159.stdout:932: write (excl) oid 17 current snap is 9 2024-09-13T13:50:38.408 INFO:tasks.rados.rados.0.smithi159.stdout:932: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-13T13:50:38.413 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.413 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.414 INFO:tasks.rados.rados.0.smithi159.stdout:932: writing smithi15938319-17 from 534489 to 1152760 tid 1 2024-09-13T13:50:38.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:38.414 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:38.419 INFO:tasks.rados.rados.0.smithi159.stdout:932: writing smithi15938319-17 from 1565566 to 2134489 tid 2 2024-09-13T13:50:38.419 INFO:tasks.rados.rados.0.smithi159.stdout:933: tier_evict oid 16 2024-09-13T13:50:38.419 INFO:tasks.rados.rados.0.smithi159.stdout:933: tier_evict oid 16 snap -1 2024-09-13T13:50:38.419 INFO:tasks.rados.rados.0.smithi159.stdout:934: set_chunk oid 5 target oid 2024-09-13T13:50:38.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:38.432 INFO:tasks.rados.rados.0.smithi159.stdout:934: set_chunk oid 5 offset: 2949632 length: 996352 target oid 1ac477f39ed30711ec0c16fb0dabd02729699d7460b23de486dfcef677f058fb offset: 0 snap -1 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:935: read oid 45 snap 6 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:935: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:936: read oid 50 snap -1 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:936: expect (ObjNum 120 snap 8 seq_num 120) 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:937: read oid 3 snap -1 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:937: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:938: write (excl) oid 49 current snap is 9 2024-09-13T13:50:38.433 INFO:tasks.rados.rados.0.smithi159.stdout:938: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-13T13:50:38.439 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:38.439 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:38.439 INFO:tasks.rados.rados.0.smithi159.stdout:938: writing smithi15938319-49 from 551296 to 1244816 tid 1 2024-09-13T13:50:38.443 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 6] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:38.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:50:38.443 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:38.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-09-13T13:50:38.444 INFO:tasks.rados.rados.0.smithi159.stdout:938: writing smithi15938319-49 from 2024048 to 2151296 tid 2 2024-09-13T13:50:38.444 INFO:tasks.rados.rados.0.smithi159.stdout:939: set_chunk oid 24 target oid 2024-09-13T13:50:38.457 INFO:tasks.rados.rados.0.smithi159.stdout:939: set_chunk oid 24 offset: 979456 length: 1574912 target oid 4054e08c30ef1d8ed04404c66e0a912e4f6693135f23205ec8bad2b4807c5b39 offset: 0 snap 6 2024-09-13T13:50:38.457 INFO:tasks.rados.rados.0.smithi159.stdout:940: set_chunk oid 38 target oid 2024-09-13T13:50:38.463 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:38.463 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:38.469 INFO:tasks.rados.rados.0.smithi159.stdout:940: set_chunk oid 38 offset: 298496 length: 1318912 target oid 2b07fbafed8d6addbf631b0bce83f56c4fcda5acd6823bf6963b63d91569eec0 offset: 0 snap 5 2024-09-13T13:50:38.470 INFO:tasks.rados.rados.0.smithi159.stdout:941: write (excl) oid 32 current snap is 9 2024-09-13T13:50:38.470 INFO:tasks.rados.rados.0.smithi159.stdout:930: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:38.470 INFO:tasks.rados.rados.0.smithi159.stdout:929: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-13T13:50:38.484 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 152897 2024-09-13T13:50:38.485 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:38.485 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:38.489 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:38.489+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:38.512 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4 4.7 4:e63fe140:::smithi15938319-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.408746440000002, 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058803479999999997, 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.531 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120164+0000", 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4499999999999997e-07 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120167+0000", 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8700000000000001e-06 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120173+0000", 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2269999999999998e-06 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.120182+0000", 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4819999999999994e-06 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121521+0000", 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001339668 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.121596+0000", 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4987000000000006e-05 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125923+0000", 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043266620000000002 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.125983+0000", 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300000000000002e-05 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.126044+0000", 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0707e-05 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:73 4.7 4:e1bd516b:::smithi15938319-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.203179306999999, 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060681499999999996, 2024-09-13T13:50:38.537 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 73 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325734+0000", 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326420+0000", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068555600000000003 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326435+0000", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.575e-05 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326444+0000", 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8410000000000006e-06 2024-09-13T13:50:38.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326517+0000", 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2784e-05 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326630+0000", 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112759 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326729+0000", 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9761999999999998e-05 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.328491+0000", 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00176133 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331735+0000", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003244317 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331768+0000", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2750000000000003e-05 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331799+0000", 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1281000000000001e-05 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.543 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:98 4.7 4:e26a40dd:::smithi15938319-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.124396645000001, 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052413879999999996, 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 98 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:38.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.404514+0000", 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2999999999999999e-07 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405146+0000", 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063201000000000002 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405157+0000", 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0773e-05 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405162+0000", 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2719999999999997e-06 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405203+0000", 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0639000000000002e-05 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405254+0000", 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0760000000000002e-05 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.405333+0000", 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9501999999999999e-05 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.406334+0000", 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010006590000000001 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409690+0000", 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033561060000000002 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409720+0000", 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9771999999999999e-05 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.409755+0000", 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5064999999999997e-05 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.613619607, 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:50:38.550 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:50:38.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:38.556 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.608820078000001, 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:50:38.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.759472039999999, 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:50:38.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.454671729000001, 2024-09-13T13:50:38.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:50:38.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:50:38.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.575 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 20.365790485000002, 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.576 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:50:38.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:38.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:50:38.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1111 4.7 4:e1bd516b:::smithi15938319-19:head [write 527753~708201,stat] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.598650162, 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060230190000000001, 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.583 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1111 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930262+0000", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1229999999999998e-06 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930922+0000", 2024-09-13T13:50:38.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065976499999999998 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930939+0000", 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7158000000000001e-05 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930948+0000", 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931023+0000", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5562000000000006e-05 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931094+0000", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0445999999999996e-05 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931198+0000", 2024-09-13T13:50:38.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010367899999999999 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.932418+0000", 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012203419999999999 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936222+0000", 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038036860000000001 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936256+0000", 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4477000000000002e-05 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936283+0000", 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6970000000000001e-05 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.589 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.255174439999999, 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:50:38.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:50:38.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:50:38.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.249314491, 2024-09-13T13:50:38.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:38.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:50:38.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.24436985, 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:50:38.605 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:50:38.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:38.613 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.244159271999999, 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:50:38.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:50:38.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:50:38.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.244111173, 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.624 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:50:38.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:38.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:50:38.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:50:38.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.161030761999999, 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.631 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:38.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.160922051, 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:50:38.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:50:38.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.160806771000001, 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.642 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:50:38.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(4.7 53/52 [PushReplyOp(4:e63fe140:::smithi15938319-1:head)])", 2024-09-13T13:50:38.647 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.4666024820000008, 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064523219999999999, 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8500000000000002e-07 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062312+0000", 2024-09-13T13:50:38.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8639999999999998e-06 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062335+0000", 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3156999999999999e-05 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062340+0000", 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0540000000000002e-06 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068597+0000", 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062569940000000001 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068760+0000", 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016276800000000001 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1328 4.7 4:e26a40dd:::smithi15938319-25:head [write 796665~544891] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:38.651 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.5794069430000004, 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085750689999999994, 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1328 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949505+0000", 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1409999999999999e-06 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950074+0000", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056815299999999995 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950091+0000", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7385999999999999e-05 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950118+0000", 2024-09-13T13:50:38.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7101e-05 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950198+0000", 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9474999999999994e-05 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950204+0000", 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4659999999999996e-06 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952911+0000", 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027069099999999999 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952989+0000", 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8485000000000005e-05 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.953127+0000", 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746799999999999 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957228+0000", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041009810000000001 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957981+0000", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075359999999999999 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958037+0000", 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5349999999999997e-05 2024-09-13T13:50:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958078+0000", 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1553e-05 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1329 4.7 4:e26a40dd:::smithi15938319-25:head [write 1759554~650689] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.955203+0000", 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.5737072380000008, 2024-09-13T13:50:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054698409999999996, 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1329 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955203+0000", 2024-09-13T13:50:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955203+0000", 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955204+0000", 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2589999999999999e-06 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955845+0000", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000640403 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955860+0000", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5328e-05 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.955868+0000", 2024-09-13T13:50:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9100000000000005e-06 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.956073+0000", 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000204837 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.956150+0000", 2024-09-13T13:50:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7558999999999994e-05 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.956275+0000", 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012520399999999999 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958093+0000", 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018177530000000001 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.960529+0000", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024356040000000001 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.960619+0000", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0518000000000003e-05 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.960673+0000", 2024-09-13T13:50:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3465999999999999e-05 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:38.665 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:38.667 INFO:tasks.rados.rados.0.smithi159.stdout:928: expect (ObjNum 111 snap 8 seq_num 111) 2024-09-13T13:50:38.667 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3809270 2024-09-13T13:50:38.667 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:38.667 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1183364 2024-09-13T13:50:38.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:38.668 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:38.665+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: oid 30 updating version 0 to 67 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: oid 30 updating version 67 to 68 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: oid 30 updating version 68 to 69 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 69 (ObjNum 125 snap 9 seq_num 125) dirty exists 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:930: left oid 30 (ObjNum 125 snap 9 seq_num 125) 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:931: finishing write tid 1 to smithi15938319-9 2024-09-13T13:50:38.693 INFO:tasks.rados.rados.0.smithi159.stdout:941: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-13T13:50:38.700 INFO:tasks.rados.rados.0.smithi159.stdout:941: writing smithi15938319-32 from 568103 to 1336872 tid 1 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:941: writing smithi15938319-32 from 2082530 to 2168103 tid 2 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:928: done (13 left) 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:929: done (12 left) 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:930: done (11 left) 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:942: delete oid 22 current snap is 9 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:931: finishing write tid 2 to smithi15938319-9 2024-09-13T13:50:38.705 INFO:tasks.rados.rados.0.smithi159.stdout:931: finishing write tid 3 to smithi15938319-9 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:931: finishing write tid 4 to smithi15938319-9 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:931: oid 9 updating version 0 to 58 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:931: oid 9 updating version 58 to 59 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:931: oid 9 updating version 59 to 60 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 60 (ObjNum 126 snap 9 seq_num 126) dirty exists 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:931: left oid 9 (ObjNum 126 snap 9 seq_num 126) 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: finishing write tid 1 to smithi15938319-17 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:933: got (22) Invalid argument 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: finishing write tid 2 to smithi15938319-17 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: finishing write tid 3 to smithi15938319-17 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: finishing write tid 4 to smithi15938319-17 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: oid 17 updating version 0 to 94 2024-09-13T13:50:38.706 INFO:tasks.rados.rados.0.smithi159.stdout:932: oid 17 updating version 94 to 96 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:932: oid 17 version 96 is already newer than 95 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 96 (ObjNum 127 snap 9 seq_num 127) dirty exists 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:932: left oid 17 (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:934: finishing set_chunk to oid 5 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 49 (ObjNum 117 snap 8 seq_num 117) dirty exists 2024-09-13T13:50:38.707 INFO:tasks.rados.rados.0.smithi159.stdout:937: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:38.730 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 435978 2024-09-13T13:50:38.747 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 58 (0%) 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:avg 3.625 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 1.53603 -> 1.53603 (expected baseline 1.84349) 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.0 with 1 -> 1 pgs (0.275862 -> 0.275862 * mean) 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 7 -> 7 pgs (1.93103 -> 1.93103 * mean) 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:38.753 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-09-13T13:50:38.754 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 3 2024-09-13T13:50:38.754 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:38.754 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0005 2024-09-13T13:50:38.754 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0054 2024-09-13T13:50:38.754 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:38.766 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:38.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-09-13T13:50:38.767 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:38.767+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:38.770 INFO:tasks.rados.rados.0.smithi159.stdout:935: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:38.791 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2720599 2024-09-13T13:50:38.810 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.811 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:38.868 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:38.868+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:38.882 INFO:tasks.rados.rados.0.smithi159.stdout:936: expect (ObjNum 120 snap 8 seq_num 120) 2024-09-13T13:50:38.882 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:38.910 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:38.922 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:38.922 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:38.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:38.966 INFO:tasks.rados.rados.0.smithi159.stdout:938: finishing write tid 1 to smithi15938319-49 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: finishing write tid 2 to smithi15938319-49 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: finishing write tid 3 to smithi15938319-49 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: finishing write tid 4 to smithi15938319-49 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: oid 49 updating version 0 to 8 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: oid 49 version 8 is already newer than 7 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: oid 49 updating version 8 to 9 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 9 (ObjNum 128 snap 9 seq_num 128) dirty exists 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:938: left oid 49 (ObjNum 128 snap 9 seq_num 128) 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:939: finishing set_chunk to oid 24 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:940: finishing set_chunk to oid 38 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 298496 length: 1318912 2024-09-13T13:50:38.967 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:931: done (11 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:932: done (10 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:933: done (9 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:934: done (8 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:935: done (7 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:936: done (6 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:937: done (5 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:938: done (4 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:939: done (3 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:940: done (2 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:942: done (1 left) 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:943: read oid 4 snap -1 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:943: expect deleted 2024-09-13T13:50:38.968 INFO:tasks.rados.rados.0.smithi159.stdout:944: read oid 12 snap -1 2024-09-13T13:50:38.969 INFO:tasks.rados.rados.0.smithi159.stdout:944: expect (ObjNum 103 snap 6 seq_num 103) 2024-09-13T13:50:38.969 INFO:tasks.rados.rados.0.smithi159.stdout:945: snap_remove snap 8 2024-09-13T13:50:38.969 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:38.969+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:39.034 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:39.034 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:39.043 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:39.086 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:39.085+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:39.113 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 58 (0%) 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:avg 3.625 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 1.53603 -> 1.53603 (expected baseline 1.84349) 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.0 with 1 -> 1 pgs (0.275862 -> 0.275862 * mean) 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 7 -> 7 pgs (1.93103 -> 1.93103 * mean) 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 1 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0005 2024-09-13T13:50:39.114 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0054 2024-09-13T13:50:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:39.126 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:39.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-09-13T13:50:39.146 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:39.146 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:39.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:39.210 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:39.210+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:39.222 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:50:39.226 INFO:tasks.rados.rados.0.smithi159.stdout:945: done (3 left) 2024-09-13T13:50:39.226 INFO:tasks.rados.rados.0.smithi159.stdout:946: read oid 6 snap -1 2024-09-13T13:50:39.226 INFO:tasks.rados.rados.0.smithi159.stdout:946: expect (ObjNum 121 snap 8 seq_num 121) 2024-09-13T13:50:39.226 INFO:tasks.rados.rados.0.smithi159.stdout:947: write oid 20 current snap is 9 2024-09-13T13:50:39.227 INFO:tasks.rados.rados.0.smithi159.stdout:941: finishing write tid 1 to smithi15938319-32 2024-09-13T13:50:39.228 INFO:tasks.rados.rados.0.smithi159.stdout:941: finishing write tid 2 to smithi15938319-32 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: finishing write tid 3 to smithi15938319-32 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: finishing write tid 4 to smithi15938319-32 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: oid 32 updating version 0 to 43 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: oid 32 updating version 43 to 44 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: oid 32 version 44 is already newer than 42 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 44 (ObjNum 129 snap 9 seq_num 129) dirty exists 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:941: left oid 32 (ObjNum 129 snap 9 seq_num 129) 2024-09-13T13:50:39.229 INFO:tasks.rados.rados.0.smithi159.stdout:947: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-13T13:50:39.232 INFO:tasks.rados.rados.0.smithi159.stdout:947: writing smithi15938319-20 from 601717 to 1237337 tid 1 2024-09-13T13:50:39.238 INFO:tasks.rados.rados.0.smithi159.stdout:947: writing smithi15938319-20 from 1799494 to 2201717 tid 2 2024-09-13T13:50:39.238 INFO:tasks.rados.rados.0.smithi159.stdout:941: done (4 left) 2024-09-13T13:50:39.238 INFO:tasks.rados.rados.0.smithi159.stdout:948: write oid 50 current snap is 9 2024-09-13T13:50:39.238 INFO:tasks.rados.rados.0.smithi159.stdout:948: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-13T13:50:39.246 INFO:tasks.rados.rados.0.smithi159.stdout:948: writing smithi15938319-50 from 618524 to 1329393 tid 1 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:948: writing smithi15938319-50 from 1857976 to 2218524 tid 2 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:949: read oid 1 snap -1 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:949: expect deleted 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:950: read oid 9 snap -1 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:950: expect (ObjNum 126 snap 9 seq_num 126) 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:951: write oid 36 current snap is 9 2024-09-13T13:50:39.251 INFO:tasks.rados.rados.0.smithi159.stdout:951: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-13T13:50:39.258 INFO:tasks.rados.rados.0.smithi159.stdout:951: writing smithi15938319-36 from 635331 to 1421449 tid 1 2024-09-13T13:50:39.262 INFO:tasks.rados.rados.0.smithi159.stdout:951: writing smithi15938319-36 from 1916458 to 2235331 tid 2 2024-09-13T13:50:39.263 INFO:tasks.rados.rados.0.smithi159.stdout:952: read oid 32 snap -1 2024-09-13T13:50:39.263 INFO:tasks.rados.rados.0.smithi159.stdout:952: expect (ObjNum 129 snap 9 seq_num 129) 2024-09-13T13:50:39.263 INFO:tasks.rados.rados.0.smithi159.stdout:944: expect (ObjNum 103 snap 6 seq_num 103) 2024-09-13T13:50:39.263 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:39.272 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:39.271+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:39.301 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:39.302 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:39.309 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:39.309 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:39.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:39.330 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:39.330 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:39.352 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:39.373 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:39.374 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:39.374 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:39.373+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:39.392 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:39.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:39.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 48.178757046999998, 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:50:39.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:50:39.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:50:39.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:39.421 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.170722900999998, 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:50:39.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:50:39.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:50:39.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:50:39.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:50:39.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.429 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.432 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.432 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:39.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:39.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:39.455 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:39.456 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:39.475 INFO:tasks.rados.rados.0.smithi159.stdout:946: expect (ObjNum 121 snap 8 seq_num 121) 2024-09-13T13:50:39.476 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:39.477 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:39.477 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:39.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:39.476+0000 7f6abb7fe640 0 -- 172.21.15.27:0/2758121493 >> v2:172.21.15.100:6800/450451503 conn(0x7f6abc21cd90 msgr2=0x7f6abc21f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:39.480 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:39.479+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:39.485 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:39.484+0000 7f26bafde640 -1 osd.3 61 handle_osd_map injecting map crc failure 2024-09-13T13:50:39.498 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:39.521 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:39.540 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:39.540 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: finishing write tid 1 to smithi15938319-20 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: finishing write tid 2 to smithi15938319-20 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: finishing write tid 3 to smithi15938319-20 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: finishing write tid 4 to smithi15938319-20 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: oid 20 updating version 0 to 55 2024-09-13T13:50:39.543 INFO:tasks.rados.rados.0.smithi159.stdout:947: oid 20 updating version 55 to 57 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:947: oid 20 version 57 is already newer than 56 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 57 (ObjNum 131 snap 9 seq_num 131) dirty exists 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:947: left oid 20 (ObjNum 131 snap 9 seq_num 131) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:948: finishing write tid 1 to smithi15938319-50 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:948: finishing write tid 2 to smithi15938319-50 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:948: finishing write tid 3 to smithi15938319-50 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:943: done (8 left) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:944: done (7 left) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:946: done (6 left) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:947: done (5 left) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:949: done (4 left) 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:953: snap_create 2024-09-13T13:50:39.544 INFO:tasks.rados.rados.0.smithi159.stdout:948: finishing write tid 4 to smithi15938319-50 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:948: oid 50 updating version 0 to 64 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:948: oid 50 updating version 64 to 66 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:948: oid 50 version 66 is already newer than 65 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 66 (ObjNum 132 snap 9 seq_num 132) dirty exists 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:948: left oid 50 (ObjNum 132 snap 9 seq_num 132) 2024-09-13T13:50:39.545 INFO:tasks.rados.rados.0.smithi159.stdout:950: expect (ObjNum 126 snap 9 seq_num 126) 2024-09-13T13:50:39.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:39.576 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:39.575+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:39.606 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1466396 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: finishing write tid 1 to smithi15938319-36 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: finishing write tid 2 to smithi15938319-36 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: finishing write tid 3 to smithi15938319-36 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: finishing write tid 4 to smithi15938319-36 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: oid 36 updating version 0 to 64 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: oid 36 version 64 is already newer than 62 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: oid 36 version 64 is already newer than 63 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 64 (ObjNum 133 snap 9 seq_num 133) dirty exists 2024-09-13T13:50:39.618 INFO:tasks.rados.rados.0.smithi159.stdout:951: left oid 36 (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:50:39.619 INFO:tasks.rados.rados.0.smithi159.stdout:952: expect (ObjNum 129 snap 9 seq_num 129) 2024-09-13T13:50:39.619 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:39.677 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:39.676+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:39.692 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:39.693 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:39.699 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:39.699 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:39.701 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:39.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:39.722 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:39.743 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:39.743 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:39.778 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:39.777+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:39.847 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:39.847 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:39.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.078164997999998, 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.848 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.849 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:50:39.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:50:39.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.077187800000001, 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.856 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:50:39.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.075433135000001, 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:50:39.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:50:39.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.075040921999999, 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.868 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:50:39.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.075003594999998, 2024-09-13T13:50:39.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:39.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:50:39.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.880 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.070205242, 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:50:39.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:50:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.886 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.054570017, 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.887 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:50:39.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:50:39.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.054463667, 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.893 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:50:39.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.898 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.046529896999999, 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.899 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:50:39.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:50:39.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:50:39.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.028267153000002, 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.905 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:50:39.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:50:39.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.028141339000001, 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.911 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:50:39.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.028014549000002, 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:50:39.917 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:50:39.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:50:39.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.946816126000002, 2024-09-13T13:50:39.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:39.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:50:39.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:50:39.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:39.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.946714674999999, 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:50:39.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:50:39.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.385361909, 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.936 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:39.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.941 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.379510366, 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:50:39.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.10270073, 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:50:39.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.1176329149999997, 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:50:39.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:50:39.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.1116116399999996, 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:50:39.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1552 3.1 3:bd95eec2:::8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338:head [write 0~1948672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:39.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.754945127, 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011182592999999999, 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1552 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:39.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090026+0000", 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:39.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092134+0000", 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021088859999999999 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092153+0000", 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8260000000000001e-05 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092162+0000", 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092260+0000", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7532000000000001e-05 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092430+0000", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017035499999999999 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092587+0000", 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015724700000000001 2024-09-13T13:50:39.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.095894+0000", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306944 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101100+0000", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052056560000000003 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101156+0000", 2024-09-13T13:50:39.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6586000000000003e-05 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101206+0000", 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9626999999999998e-05 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:39.968 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:39.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:39.970 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:39.969+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:40.070 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:40.070+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:40.091 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.091 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:40.091 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:40.091 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:40.171 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:40.170+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:40.206 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.206 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:40.206 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:40.206 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:40.206 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:moved 10 / 58 (17.2414%) 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:avg 3.625 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 1.53603 -> 0.780625 (expected baseline 1.84349) 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.0 with 1 -> 4 pgs (0.275862 -> 1.10345 * mean) 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.6 with 7 -> 3 pgs (1.93103 -> 0.827586 * mean) 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:40.230 INFO:teuthology.orchestra.run.smithi027.stdout:oload 110 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 1 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0005 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0006 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 weight 1.0000 -> 0.6426 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8 weight 1.0000 -> 0.6708 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6 weight 1.0000 -> 0.6997 2024-09-13T13:50:40.231 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13 weight 1.0000 -> 0.7061 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:948: done (4 left) 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:950: done (3 left) 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:951: done (2 left) 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:952: done (1 left) 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:953: done (0 left) 2024-09-13T13:50:40.234 INFO:tasks.rados.rados.0.smithi159.stdout:954: set_chunk oid 45 target oid 2024-09-13T13:50:40.236 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:40.236+0000 7fe857f4f640 -1 osd.7 62 handle_osd_map injecting map crc failure 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:954: set_chunk oid 45 offset: 1923072 length: 1218048 target oid 1620591b66fd1488ef124d1180f72e46d491299b08178f5bbff3a43821678b56 offset: 0 snap -1 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:955: read oid 37 snap -1 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:955: expect deleted 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:956: read oid 50 snap -1 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:956: expect (ObjNum 132 snap 9 seq_num 132) 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:957: read oid 39 snap -1 2024-09-13T13:50:40.247 INFO:tasks.rados.rados.0.smithi159.stdout:957: expect deleted 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:958: tier_evict oid 19 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:958: tier_evict oid 19 snap -1 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:959: tier_evict oid 12 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:959: tier_evict oid 12 snap -1 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:960: snap_create 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:958: got (22) Invalid argument 2024-09-13T13:50:40.248 INFO:tasks.rados.rados.0.smithi159.stdout:959: got (22) Invalid argument 2024-09-13T13:50:40.249 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T13:50:40.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-09-13T13:50:40.250 INFO:tasks.rados.rados.0.smithi159.stdout:954: finishing set_chunk to oid 45 2024-09-13T13:50:40.250 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 20 (ObjNum 86 snap 5 seq_num 86) dirty exists 2024-09-13T13:50:40.252 INFO:tasks.rados.rados.0.smithi159.stdout:956: expect (ObjNum 132 snap 9 seq_num 132) 2024-09-13T13:50:40.272 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2070637 2024-09-13T13:50:40.272 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:40.271+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:40.320 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.10.22:7 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.569991842, 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072188460000000001, 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.321 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4400000000000001e-07 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749188+0000", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.644e-06 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749258+0000", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9744000000000002e-05 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749263+0000", 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6339999999999998e-06 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752867+0000", 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036041469999999998 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752952+0000", 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4035000000000004e-05 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756404+0000", 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034522979999999999 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:40.325 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.421011324999998, 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:50:40.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.331 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:265 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.407732018000001, 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082114400000000004, 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 265 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911447+0000", 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911457+0000", 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0433e-05 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911476+0000", 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8612000000000001e-05 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911486+0000", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911546+0000", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9877000000000003e-05 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911627+0000", 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1546999999999997e-05 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917585+0000", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059573229999999996 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918841+0000", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001255969 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919557+0000", 2024-09-13T13:50:40.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000716079 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919613+0000", 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5881000000000003e-05 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919656+0000", 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3643999999999998e-05 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.337 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.405383709999999, 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:50:40.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.396672035000002, 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.344 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:50:40.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.549126072, 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:50:40.350 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:50:40.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:366 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.244256331999999, 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079177420000000002, 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.356 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 366 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074921+0000", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074925+0000", 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8010000000000001e-06 2024-09-13T13:50:40.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1880000000000002e-06 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074939+0000", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075137+0000", 2024-09-13T13:50:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197927 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075206+0000", 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8806999999999994e-05 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.078002+0000", 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002796078 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.081901+0000", 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003899713 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082743+0000", 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084112100000000001 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082793+0000", 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0359999999999999e-05 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082838+0000", 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5321000000000002e-05 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:378 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.243214314999999, 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071838889999999997, 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.362 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 378 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075964+0000", 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0160000000000001e-06 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075970+0000", 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9090000000000003e-06 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075981+0000", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1042e-05 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075990+0000", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2539999999999992e-06 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076042+0000", 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1999000000000001e-05 2024-09-13T13:50:40.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076116+0000", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4060000000000006e-05 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.079664+0000", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035479420000000001 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082307+0000", 2024-09-13T13:50:40.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002643472 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083060+0000", 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075247599999999997 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083109+0000", 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9387e-05 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083146+0000", 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7332000000000001e-05 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1038 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.539153038999999, 2024-09-13T13:50:40.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078733310000000008, 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1038 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:40.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780025+0000", 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.238e-06 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780029+0000", 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1470000000000003e-06 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780040+0000", 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0597e-05 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780049+0000", 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3260000000000002e-06 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780132+0000", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3185999999999999e-05 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780189+0000", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7234000000000001e-05 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.787897+0000", 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077076030000000004 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.372 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1246 4.1 4:81ab2613:::smithi15938319-29:head [write 2873212~588997] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.067615467, 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067622710000000003, 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.373 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1246 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251561+0000", 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.251563+0000", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.767e-06 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252186+0000", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062314599999999999 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252194+0000", 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1929999999999993e-06 2024-09-13T13:50:40.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252200+0000", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0839999999999999e-06 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252247+0000", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7852000000000002e-05 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252301+0000", 2024-09-13T13:50:40.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4085000000000001e-05 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252403+0000", 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101209 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258157+0000", 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057543159999999998 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258206+0000", 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8809e-05 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258285+0000", 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8881999999999995e-05 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258324+0000", 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8927999999999997e-05 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.378 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1247 4.1 4:81ab2613:::smithi15938319-29:head [setxattr _header (56) in=7b,truncate 3462209] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.066965882, 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006237584, 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.379 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1247 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252211+0000", 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252212+0000", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.06e-07 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252214+0000", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8480000000000001e-06 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252223+0000", 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4069999999999993e-06 2024-09-13T13:50:40.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252227+0000", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.109e-06 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252920+0000", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00069335899999999995 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252981+0000", 2024-09-13T13:50:40.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1035000000000001e-05 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253083+0000", 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101807 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258162+0000", 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00507874 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258375+0000", 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021354999999999999 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258410+0000", 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4857999999999997e-05 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258449+0000", 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8265e-05 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1248 4.1 4:81ab2613:::smithi15938319-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.066936917, 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063470940000000002, 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.385 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1248 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252240+0000", 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0999999999999999e-07 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252243+0000", 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2500000000000001e-06 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252248+0000", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.609e-06 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252254+0000", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4330000000000003e-06 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253282+0000", 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010284160000000001 2024-09-13T13:50:40.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.253338+0000", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6440000000000002e-05 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258482+0000", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005143997 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258532+0000", 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9527999999999997e-05 2024-09-13T13:50:40.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258587+0000", 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5010999999999999e-05 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1251 4.1 4:80c58c2d:::smithi15938319-35:head [read 649379~1922059,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.066892563, 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067450039999999998, 2024-09-13T13:50:40.390 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1251 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252284+0000", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-07 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252286+0000", 2024-09-13T13:50:40.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.452e-06 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252289+0000", 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.196e-06 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.252292+0000", 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6129999999999998e-06 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258607+0000", 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063152319999999996 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.258663+0000", 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5918999999999998e-05 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.259029+0000", 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000366375 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 11.572385884999999, 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:50:40.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing(4.f epoch 53/34 rep_tid 118 entries 53'57 (0'0) error 4:f883ee18:::smithi15938319-40:head by client.4571.0:1290 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 51'54)", 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 11.569549879, 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084851660000000006, 2024-09-13T13:50:40.399 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749628+0000", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0400000000000005e-07 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749637+0000", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749672+0000", 2024-09-13T13:50:40.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4409999999999998e-05 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749685+0000", 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2754e-05 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754728+0000", 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050430300000000004 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758112+0000", 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033844600000000002 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 10.357715605999999, 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.403 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:50:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:50:40.408 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.3559139800000004, 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:50:40.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:1512 4.f e58/34 4:f2f09040:::smithi15938319-31:head v 58'59, mlcod=53'56)", 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:33.737435+0000", 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.5817414469999997, 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006345814, 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737435+0000", 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737435+0000", 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737436+0000", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.84e-07 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737924+0000", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00048785799999999998 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737947+0000", 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3825999999999999e-05 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.737957+0000", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4329999999999993e-06 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.738019+0000", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2367999999999996e-05 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.738051+0000", 2024-09-13T13:50:40.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.743733+0000", 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005682243 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:33.743781+0000", 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8321e-05 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1535 4.1 4:80c58c2d:::smithi15938319-35:9 [sparse-read 2363141~73149] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:34.726714+0000", 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5924628720000005, 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059275969999999997, 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1535 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.418 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726714+0000", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726714+0000", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726714+0000", 2024-09-13T13:50:40.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.63e-07 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726717+0000", 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.413e-06 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726722+0000", 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.726727+0000", 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.732418+0000", 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0056908990000000001 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.732508+0000", 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0394000000000002e-05 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.732642+0000", 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000133282 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:1757 4.e e62/34 4:709ad093:::smithi15938319-50:head v 62'64, mlcod=61'61)", 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:39.246993+0000", 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.072184222, 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066414539999999998, 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.422 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.246993+0000", 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.246993+0000", 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.246993+0000", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8100000000000002e-07 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.248007+0000", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001014105 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.248060+0000", 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2510999999999999e-05 2024-09-13T13:50:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.248066+0000", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.818e-06 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.248152+0000", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6007999999999994e-05 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.248170+0000", 2024-09-13T13:50:40.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8130000000000001e-05 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.253595+0000", 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054250670000000004 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:39.253634+0000", 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9033999999999999e-05 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.427 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.427 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.428 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:40.429 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:40.427+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:40.529 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:40.529+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:40.532 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.532 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:40.532 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:40.532 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 58 (0%) 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:avg 3.625 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 0.780625 -> 0.780625 (expected baseline 1.84349) 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.3 with 2 -> 2 pgs (0.551724 -> 0.551724 * mean) 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.2 with 5 -> 5 pgs (1.37931 -> 1.37931 * mean) 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-09-13T13:50:40.602 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 3 2024-09-13T13:50:40.603 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:40.603 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0005 2024-09-13T13:50:40.603 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0054 2024-09-13T13:50:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:40.615 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T13:50:40.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-09-13T13:50:40.630 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:40.629+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:40.641 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.642 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:40.642 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:40.642 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:40.642 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:40.731 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:40.730+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 52.212128857000003, 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.756 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:50:40.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 50.157802967999999, 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.762 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:50:40.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:50:40.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:50:40.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:40.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:50:40.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 50.144424577999999, 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:50:40.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.331091641, 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:50:40.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:50:40.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525679633999999, 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.780 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:50:40.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:50:40.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.785 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525642462, 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:50:40.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:50:40.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:50:40.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525621734, 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.792 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:50:40.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:40.797 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525525433, 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:40.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:50:40.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:50:40.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:40.803 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525502789000001, 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:40.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:50:40.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:50:40.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.809 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.525476759, 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:40.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.385975891999999, 2024-09-13T13:50:40.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:50:40.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:50:40.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:40.821 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.385868945, 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:50:40.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:50:40.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.385839686000001, 2024-09-13T13:50:40.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:40.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:50:40.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:50:40.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.833 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.379475283, 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:50:40.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:50:40.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:50:40.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.374515089000001, 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:50:40.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1279 4.4 4:22f57a3c:::smithi15938319-42:head [write 2555481~425187] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.369670957, 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.37714365399999999, 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1279 2024-09-13T13:50:40.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384129+0000", 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384523+0000", 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039437499999999998 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384535+0000", 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1626e-05 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384541+0000", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4699999999999999e-06 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1349e-05 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384589+0000", 2024-09-13T13:50:40.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3249999999999996e-06 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728981+0000", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34439230599999998 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729015+0000", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4149000000000001e-05 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740625+0000", 2024-09-13T13:50:40.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011609895 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740676+0000", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.13e-05 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754660+0000", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013983423 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754774+0000", 2024-09-13T13:50:40.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114554 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754954+0000", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017924600000000001 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759913+0000", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049594189999999996 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761144+0000", 2024-09-13T13:50:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001231174 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761182+0000", 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8235999999999998e-05 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761271+0000", 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8882999999999996e-05 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1280 4.4 4:22f57a3c:::smithi15938319-42:head [setxattr _header (56) in=7b,truncate 3512630] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.36924396, 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379144494, 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1280 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.858 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:50:40.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.99e-07 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384559+0000", 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384564+0000", 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0610000000000003e-06 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384569+0000", 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384617+0000", 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8479000000000001e-05 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384623+0000", 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.677e-06 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729055+0000", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34443185999999998 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729104+0000", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9335000000000001e-05 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740720+0000", 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011616191 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740766+0000", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5667000000000002e-05 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759391+0000", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018624900999999999 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759450+0000", 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8891000000000002e-05 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759563+0000", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011324600000000001 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761528+0000", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019654220000000001 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763564+0000", 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020357410000000002 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763611+0000", 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6873000000000003e-05 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763699+0000", 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8288000000000006e-05 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1281 4.4 4:22f57a3c:::smithi15938319-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:40.865 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.369216582, 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379374818, 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1281 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:40.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1899999999999998e-07 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384585+0000", 2024-09-13T13:50:40.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.334e-06 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384591+0000", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8259999999999998e-06 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384596+0000", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6439999999999997e-06 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384640+0000", 2024-09-13T13:50:40.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3217999999999999e-05 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384643+0000", 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6330000000000002e-06 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729130+0000", 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34448669500000001 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:40.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729160+0000", 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0035999999999998e-05 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740807+0000", 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011646844 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740902+0000", 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4952999999999997e-05 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760235+0000", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019333605 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760386+0000", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015056100000000001 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763739+0000", 2024-09-13T13:50:40.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033526620000000002 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763822+0000", 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3255000000000002e-05 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763957+0000", 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013513300000000001 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4325253719999997, 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:50:40.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:50:40.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:40.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.878 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4324862339999997, 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:40.879 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:50:40.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:40.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:40.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:40.884 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:40.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:40.886 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:40.885+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:moved 0 / 58 (0%) 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:avg 3.625 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:stddev 0.780625 -> 0.780625 (expected baseline 1.84349) 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:min osd.3 with 2 -> 2 pgs (0.551724 -> 0.551724 * mean) 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:max osd.2 with 5 -> 5 pgs (1.37931 -> 1.37931 * mean) 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:50:40.917 INFO:teuthology.orchestra.run.smithi027.stdout:oload 1000 2024-09-13T13:50:40.918 INFO:teuthology.orchestra.run.smithi027.stdout:max_change 3 2024-09-13T13:50:40.918 INFO:teuthology.orchestra.run.smithi027.stdout:max_change_osds 4 2024-09-13T13:50:40.918 INFO:teuthology.orchestra.run.smithi027.stdout:average_utilization 0.0005 2024-09-13T13:50:40.918 INFO:teuthology.orchestra.run.smithi027.stdout:overload_utilization 0.0054 2024-09-13T13:50:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:no change 2024-09-13T13:50:40.986 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:40.985+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:41.029 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.029 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:41.029 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:41.029 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.038 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:41.087 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:41.086+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:41.140 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.140 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:41.140 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:41.140 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:41.140 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:41.190 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:41.189+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:41.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:50:41.242 INFO:tasks.rados.rados.0.smithi159.stdout:954: done (6 left) 2024-09-13T13:50:41.242 INFO:tasks.rados.rados.0.smithi159.stdout:956: done (5 left) 2024-09-13T13:50:41.242 INFO:tasks.rados.rados.0.smithi159.stdout:957: done (4 left) 2024-09-13T13:50:41.242 INFO:tasks.rados.rados.0.smithi159.stdout:958: done (3 left) 2024-09-13T13:50:41.242 INFO:tasks.rados.rados.0.smithi159.stdout:959: done (2 left) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:960: done (1 left) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:961: read oid 8 snap 6 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:961: expect (ObjNum 94 snap 6 seq_num 94) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:962: read oid 36 snap 9 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:962: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:963: read oid 1 snap -1 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:963: expect deleted 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:964: read oid 5 snap -1 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:964: expect (ObjNum 117 snap 8 seq_num 117) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:965: read oid 48 snap 10 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:965: expect (ObjNum 104 snap 6 seq_num 104) 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:966: read oid 33 snap 5 2024-09-13T13:50:41.243 INFO:tasks.rados.rados.0.smithi159.stdout:966: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:41.244 INFO:tasks.rados.rados.0.smithi159.stdout:967: write (excl) oid 28 current snap is 11 2024-09-13T13:50:41.244 INFO:tasks.rados.rados.0.smithi159.stdout:967: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-13T13:50:41.248 INFO:tasks.rados.rados.0.smithi159.stdout:967: writing smithi15938319-28 from 652138 to 1113505 tid 1 2024-09-13T13:50:41.255 INFO:tasks.rados.rados.0.smithi159.stdout:967: writing smithi15938319-28 from 1574940 to 2252138 tid 2 2024-09-13T13:50:41.255 INFO:tasks.rados.rados.0.smithi159.stdout:968: read oid 9 snap 7 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:968: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:969: tier_promote oid 3 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:970: read oid 22 snap -1 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:970: expect deleted 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:963: done (10 left) 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:971: write (excl) oid 49 current snap is 11 2024-09-13T13:50:41.256 INFO:tasks.rados.rados.0.smithi159.stdout:971: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-13T13:50:41.256 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:41.255+0000 7f002fbdf640 -1 osd.15 63 handle_osd_map injecting map crc failure 2024-09-13T13:50:41.262 INFO:tasks.rados.rados.0.smithi159.stdout:971: writing smithi15938319-49 from 668945 to 1205561 tid 1 2024-09-13T13:50:41.268 INFO:tasks.rados.rados.0.smithi159.stdout:971: writing smithi15938319-49 from 1633422 to 2124785 tid 2 2024-09-13T13:50:41.268 INFO:tasks.rados.rados.0.smithi159.stdout:972: set_chunk oid 46 target oid 2024-09-13T13:50:41.281 INFO:tasks.rados.rados.0.smithi159.stdout:972: set_chunk oid 46 offset: 728064 length: 822272 target oid 76aaac68466283e13d1d9d12e71eb7444a4bc40252244d2a2f1c283db3655dee offset: 0 snap -1 2024-09-13T13:50:41.281 INFO:tasks.rados.rados.0.smithi159.stdout:973: rollback oid 7 current snap is 11 2024-09-13T13:50:41.281 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 7 to 10 2024-09-13T13:50:41.281 INFO:tasks.rados.rados.0.smithi159.stdout:968: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-13T13:50:41.288 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:41.288+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:41.308 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.569809631999998, 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.309 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:50:41.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:50:41.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.314 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564729105000001, 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024188794, 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.315 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742196+0000", 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742198+0000", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3909999999999999e-06 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742205+0000", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742227+0000", 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2688999999999999e-05 2024-09-13T13:50:41.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742237+0000", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0151e-05 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742312+0000", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4913999999999995e-05 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for blocked object", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742364+0000", 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1774999999999998e-05 2024-09-13T13:50:41.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756421+0000", 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014057106999999999 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756508+0000", 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6533999999999995e-05 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766385+0000", 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098774239999999992 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.56456901, 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:50:41.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:50:41.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:50:41.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564517695999999, 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.326 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:50:41.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:50:41.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:50:41.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564445499000001, 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:50:41.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564406494, 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.338 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:50:41.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:213 3.3 3:d7665af5:::smithi15938319-14 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564378983000001, 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024017457999999998, 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.344 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 213 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742546+0000", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742547+0000", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6800000000000001e-07 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742550+0000", 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.965e-06 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742570+0000", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0207000000000002e-05 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742576+0000", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8889999999999999e-06 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743625+0000", 2024-09-13T13:50:41.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001049407 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743730+0000", 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105025 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.751509+0000", 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077789039999999997 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766480+0000", 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014970343000000001 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766513+0000", 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3009000000000002e-05 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766540+0000", 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766564+0000", 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3799e-05 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.564342082, 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:50:41.350 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:50:41.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:201 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.551006558000001, 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010542467, 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755919+0000", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755920+0000", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.513e-06 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755926+0000", 2024-09-13T13:50:41.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5509999999999999e-06 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755953+0000", 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6687999999999999e-05 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755972+0000", 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9550000000000001e-05 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766321+0000", 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010349205 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766329+0000", 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3479999999999998e-06 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766461+0000", 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132612 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.538991985999999, 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.361 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:50:41.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.366 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:203 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.532484508, 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019371698999999999, 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:41.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774441+0000", 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5199999999999998e-07 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774444+0000", 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5600000000000001e-06 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793662+0000", 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019218452 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793669+0000", 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793716+0000", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6953999999999998e-05 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793722+0000", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793813+0000", 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0705999999999995e-05 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.370 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:221 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.513285473, 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018779990999999999, 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793640+0000", 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793641+0000", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.316e-06 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793646+0000", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7870000000000001e-06 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812292+0000", 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018646276 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812298+0000", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.152e-06 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812336+0000", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8336e-05 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812344+0000", 2024-09-13T13:50:41.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3909999999999998e-06 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812420+0000", 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5733000000000001e-05 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:234 3.3 3:cdbb9bf3:::smithi15938319-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.508587279, 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021808134999999999, 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 234 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798338+0000", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798339+0000", 2024-09-13T13:50:41.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1990000000000001e-06 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798344+0000", 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.831e-06 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798354+0000", 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8670000000000006e-06 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798359+0000", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0309999999999998e-06 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798399+0000", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0191999999999997e-05 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798514+0000", 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114484 2024-09-13T13:50:41.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.803569+0000", 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005055492 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808174+0000", 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046051089999999996 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820010+0000", 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011835215 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820079+0000", 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9377999999999999e-05 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820146+0000", 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7336999999999996e-05 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.381 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.508431162000001, 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.382 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:50:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:50:41.387 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.487104575, 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:50:41.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:50:41.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.408888298000001, 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.392 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:50:41.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:250 3.3 3:edb81fa6:::smithi15938319-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.408818192999998, 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019084286999999998, 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.398 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 250 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898107+0000", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898108+0000", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6699999999999999e-07 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898111+0000", 2024-09-13T13:50:41.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8710000000000001e-06 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898131+0000", 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9939e-05 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898136+0000", 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8599999999999998e-06 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898476+0000", 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033951499999999998 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898641+0000", 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016495999999999999 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909561+0000", 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010919988 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.916236+0000", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066746660000000001 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917085+0000", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084985200000000001 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917156+0000", 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0246999999999993e-05 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917192+0000", 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5921999999999999e-05 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.6.0:178 4.5 e45/34 4:a1652551:::smithi15938319-33:head v 45'26, mlcod=42'24)", 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.657431611, 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085953929999999998, 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.404 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649494+0000", 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.649495+0000", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5600000000000004e-07 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651055+0000", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015604760000000001 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651121+0000", 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5664e-05 2024-09-13T13:50:41.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651135+0000", 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4263e-05 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651207+0000", 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2053999999999998e-05 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.651244+0000", 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7113999999999997e-05 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.658045+0000", 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0068001379999999998 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.658089+0000", 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4728000000000003e-05 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.408 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1149 4.5 4:a1652551:::smithi15938319-33:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.322770607999999, 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016672093999999998, 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.409 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1149 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984155+0000", 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984157+0000", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2819999999999999e-06 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984165+0000", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7500000000000003e-06 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984182+0000", 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7207e-05 2024-09-13T13:50:41.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984192+0000", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0237999999999999e-05 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984249+0000", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6409999999999997e-05 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:24.984311+0000", 2024-09-13T13:50:41.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2706999999999994e-05 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:25.000827+0000", 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016515499999999999 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.14.23:52 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:34.738748+0000", 2024-09-13T13:50:41.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.5681770960000003, 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085428009999999992, 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738748+0000", 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738748+0000", 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738750+0000", 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9319999999999999e-06 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738758+0000", 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8860000000000004e-06 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738786+0000", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7542000000000002e-05 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738795+0000", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6290000000000005e-06 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738866+0000", 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0437000000000003e-05 2024-09-13T13:50:41.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.738872+0000", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.745328+0000", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064562209999999998 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.745409+0000", 2024-09-13T13:50:41.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0557999999999997e-05 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.745560+0000", 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015087100000000001 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.746849+0000", 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012892979999999999 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.747236+0000", 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00038725800000000002 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.747268+0000", 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1653999999999999e-05 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:34.747291+0000", 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3363999999999999e-05 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.420 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.420 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.420 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.421 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:41.421 INFO:tasks.rados.rados.0.smithi159.stdout:969: got (0) Success 2024-09-13T13:50:41.421 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 62 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:41.421 INFO:tasks.rados.rados.0.smithi159.stdout:961: expect (ObjNum 94 snap 6 seq_num 94) 2024-09-13T13:50:41.422 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:41.422 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:41.421+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:41.512 INFO:tasks.rados.rados.0.smithi159.stdout:971: finishing write tid 1 to smithi15938319-49 2024-09-13T13:50:41.512 INFO:tasks.rados.rados.0.smithi159.stdout:965: expect (ObjNum 104 snap 6 seq_num 104) 2024-09-13T13:50:41.522 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:41.522+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:41.553 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3113149 2024-09-13T13:50:41.565 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.565 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:41.565 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:41.565 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:41.623 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:41.623+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:41.637 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:41.685 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.685 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:41.685 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:50:41.685 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:50:41.685 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: finishing write tid 2 to smithi15938319-49 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: finishing write tid 3 to smithi15938319-49 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: finishing write tid 4 to smithi15938319-49 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: oid 49 updating version 0 to 15 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: oid 49 version 15 is already newer than 13 2024-09-13T13:50:41.708 INFO:tasks.rados.rados.0.smithi159.stdout:971: oid 49 version 15 is already newer than 14 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 15 (ObjNum 135 snap 11 seq_num 135) dirty exists 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:971: left oid 49 (ObjNum 135 snap 11 seq_num 135) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:955: done (12 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:961: done (11 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:965: done (10 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:968: done (9 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:969: done (8 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:970: done (7 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:971: done (6 left) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:974: read oid 34 snap -1 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:974: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:975: write (excl) oid 9 current snap is 11 2024-09-13T13:50:41.709 INFO:tasks.rados.rados.0.smithi159.stdout:967: finishing write tid 1 to smithi15938319-28 2024-09-13T13:50:41.710 INFO:tasks.rados.rados.0.smithi159.stdout:972: finishing set_chunk to oid 46 2024-09-13T13:50:41.710 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 728064 length: 822272 2024-09-13T13:50:41.710 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T13:50:41.710 INFO:tasks.rados.rados.0.smithi159.stdout:967: finishing write tid 2 to smithi15938319-28 2024-09-13T13:50:41.710 INFO:tasks.rados.rados.0.smithi159.stdout:964: expect (ObjNum 117 snap 8 seq_num 117) 2024-09-13T13:50:41.724 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:41.723+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:41.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:41.799 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:41.799 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:41.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 51.569470076999998, 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:50:41.800 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:50:41.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.806 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.064731895000001, 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.807 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:50:41.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.064682583, 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.813 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:50:41.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:50:41.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.064666879000001, 2024-09-13T13:50:41.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:41.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:41.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:50:41.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.825 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.064639202999999, 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:50:41.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.029421605, 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:50:41.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:50:41.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:50:41.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:50:41.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.022352623, 2024-09-13T13:50:41.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:50:41.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:50:41.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:41.844 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.987814160999999, 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:41.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:50:41.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.850 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.980184318999999, 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:50:41.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.976807947000001, 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:50:41.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:50:41.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:50:41.862 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.873948667000001, 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:50:41.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.662925127000001, 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.869 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:41.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.873 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.651409844, 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:50:41.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.550101376, 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:50:41.878 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:50:41.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.550037028, 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.884 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.550008966, 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.892 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:50:41.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.897 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.54832707599999997, 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.898 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:50:41.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1809 4.2 4:4d382039:::smithi15938319-28:head [write 1574940~677198] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255103+0000", 2024-09-13T13:50:41.905 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.54199570200000002, 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026909064999999999, 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1809 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255103+0000", 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255103+0000", 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255105+0000", 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.249e-06 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255753+0000", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064809600000000002 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255768+0000", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5053e-05 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255776+0000", 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:50:41.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3489000000000001e-05 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255908+0000", 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8740999999999998e-05 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260567+0000", 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046586819999999999 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260625+0000", 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8128000000000003e-05 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.278819+0000", 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018193724000000001 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.278880+0000", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1607999999999997e-05 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.278983+0000", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010252 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280437+0000", 2024-09-13T13:50:41.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014543399999999999 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281815+0000", 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013776109999999999 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281884+0000", 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9426999999999998e-05 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.282012+0000", 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128114 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1810 4.2 4:4d382039:::smithi15938319-28:head [setxattr _header (57) in=7b,truncate 2252138] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.54130068799999997, 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030247828000000001, 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.913 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1810 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255799+0000", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4399999999999999e-07 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255804+0000", 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4909999999999999e-06 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255811+0000", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9970000000000001e-06 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255816+0000", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255947+0000", 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000130171 2024-09-13T13:50:41.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255996+0000", 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9233000000000001e-05 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260677+0000", 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046811470000000001 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:41.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260726+0000", 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9441000000000003e-05 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.279950+0000", 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019223711000000001 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280000+0000", 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9765000000000002e-05 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280101+0000", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010101499999999999 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.282160+0000", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020587470000000001 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.285789+0000", 2024-09-13T13:50:41.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036294489999999999 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.285900+0000", 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110964 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286046+0000", 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014613300000000001 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1811 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.541274056, 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030548815999999999, 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:41.921 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1811 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255828+0000", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255835+0000", 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2119999999999997e-06 2024-09-13T13:50:41.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256027+0000", 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018704600000000001 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256075+0000", 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260766+0000", 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046910119999999996 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260793+0000", 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6594999999999999e-05 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280369+0000", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019576296 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280418+0000", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9221999999999997e-05 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286090+0000", 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005671873 2024-09-13T13:50:41.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286195+0000", 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010464299999999999 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286374+0000", 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179118 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:41.927 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:41.928 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:41.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:41.929 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:41.930 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:41.931 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:41.931 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:41.932 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:41.928+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:41.934 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3115490 2024-09-13T13:50:41.934 INFO:tasks.rados.rados.0.smithi159.stdout:967: finishing write tid 3 to smithi15938319-28 2024-09-13T13:50:41.934 INFO:tasks.rados.rados.0.smithi159.stdout:967: finishing write tid 4 to smithi15938319-28 2024-09-13T13:50:41.934 INFO:tasks.rados.rados.0.smithi159.stdout:967: oid 28 updating version 0 to 69 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:967: oid 28 updating version 69 to 70 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:967: oid 28 version 70 is already newer than 68 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 70 (ObjNum 134 snap 11 seq_num 134) dirty exists 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:967: left oid 28 (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:973: finishing rollback tid 0 to smithi15938319-7 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:975: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:975: writing smithi15938319-9 from 685752 to 1297617 tid 1 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:975: writing smithi15938319-9 from 1975551 to 2285752 tid 2 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:964: done (7 left) 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:967: done (6 left) 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:972: done (5 left) 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:976: read oid 35 snap -1 2024-09-13T13:50:41.935 INFO:tasks.rados.rados.0.smithi159.stdout:976: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:977: read oid 38 snap -1 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:977: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:978: rollback oid 43 current snap is 11 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 43 to 6 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:973: finishing rollback tid 1 to smithi15938319-7 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:973: finishing rollback tid 2 to smithi15938319-7 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 75 (ObjNum 80 snap 3 seq_num 80) dirty exists 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:973: done (7 left) 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:979: read oid 45 snap -1 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:979: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:980: read oid 47 snap -1 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:980: expect (ObjNum 115 snap 8 seq_num 115) 2024-09-13T13:50:41.936 INFO:tasks.rados.rados.0.smithi159.stdout:981: read oid 14 snap -1 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:981: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:982: read oid 10 snap -1 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:982: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:983: read oid 26 snap -1 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:983: expect deleted 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:984: read oid 21 snap -1 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:984: expect (ObjNum 110 snap 8 seq_num 110) 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:985: write (excl) oid 12 current snap is 11 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:962: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1988192 2024-09-13T13:50:41.937 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:41.938 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:41.945 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:41.945 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:41.967 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:41.972 INFO:tasks.rados.rados.0.smithi159.stdout:966: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T13:50:41.975 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1652225 2024-09-13T13:50:41.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:41.988 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:42.011 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 198123 2024-09-13T13:50:42.014 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:42.029 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:42.028+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:42.037 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:42.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:42.038 INFO:tasks.rados.rados.0.smithi159.stdout:974: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:42.038 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.042 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.042 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.051 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:42.057 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:42.078 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:42.078 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:42.100 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:42.122 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:42.130 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:42.130+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:42.144 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:42.145 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:42.165 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:42.165 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:42.187 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:42.188+0000 7f77027fc640 0 -- 172.21.15.27:0/3707504135 >> v2:172.21.15.122:6808/1255553857 conn(0x7f770421f860 msgr2=0x7f7704221c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:50:42.196 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.196 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: finishing write tid 1 to smithi15938319-9 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: finishing write tid 2 to smithi15938319-9 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: finishing write tid 3 to smithi15938319-9 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: finishing write tid 4 to smithi15938319-9 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: oid 9 updating version 0 to 69 2024-09-13T13:50:42.224 INFO:tasks.rados.rados.0.smithi159.stdout:975: oid 9 updating version 69 to 70 2024-09-13T13:50:42.225 INFO:tasks.rados.rados.0.smithi159.stdout:975: oid 9 version 70 is already newer than 68 2024-09-13T13:50:42.225 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 70 (ObjNum 136 snap 11 seq_num 136) dirty exists 2024-09-13T13:50:42.225 INFO:tasks.rados.rados.0.smithi159.stdout:975: left oid 9 (ObjNum 136 snap 11 seq_num 136) 2024-09-13T13:50:42.225 INFO:tasks.rados.rados.0.smithi159.stdout:978: finishing rollback tid 0 to smithi15938319-43 2024-09-13T13:50:42.225 INFO:tasks.rados.rados.0.smithi159.stdout:985: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-13T13:50:42.230 INFO:tasks.rados.rados.0.smithi159.stdout:985: writing smithi15938319-12 from 702559 to 1106026 tid 1 2024-09-13T13:50:42.231 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:42.230+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:42.236 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:985: writing smithi15938319-12 from 1750386 to 2245418 tid 2 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:962: done (13 left) 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:966: done (12 left) 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:974: done (11 left) 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:975: done (10 left) 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:983: done (9 left) 2024-09-13T13:50:42.237 INFO:tasks.rados.rados.0.smithi159.stdout:986: write (excl) oid 31 current snap is 11 2024-09-13T13:50:42.238 INFO:tasks.rados.rados.0.smithi159.stdout:978: finishing rollback tid 1 to smithi15938319-43 2024-09-13T13:50:42.238 INFO:tasks.rados.rados.0.smithi159.stdout:978: finishing rollback tid 2 to smithi15938319-43 2024-09-13T13:50:42.238 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 68 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T13:50:42.238 INFO:tasks.rados.rados.0.smithi159.stdout:986: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-13T13:50:42.243 INFO:tasks.rados.rados.0.smithi159.stdout:986: writing smithi15938319-31 from 719366 to 1198082 tid 1 2024-09-13T13:50:42.249 INFO:tasks.rados.rados.0.smithi159.stdout:986: writing smithi15938319-31 from 1808868 to 2319366 tid 2 2024-09-13T13:50:42.249 INFO:tasks.rados.rados.0.smithi159.stdout:978: done (9 left) 2024-09-13T13:50:42.249 INFO:tasks.rados.rados.0.smithi159.stdout:987: write (excl) oid 16 current snap is 11 2024-09-13T13:50:42.249 INFO:tasks.rados.rados.0.smithi159.stdout:987: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-13T13:50:42.256 INFO:tasks.rados.rados.0.smithi159.stdout:987: writing smithi15938319-16 from 736173 to 1290138 tid 1 2024-09-13T13:50:42.262 INFO:tasks.rados.rados.0.smithi159.stdout:987: writing smithi15938319-16 from 1867350 to 2336173 tid 2 2024-09-13T13:50:42.262 INFO:tasks.rados.rados.0.smithi159.stdout:988: write oid 24 current snap is 11 2024-09-13T13:50:42.262 INFO:tasks.rados.rados.0.smithi159.stdout:988: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-13T13:50:42.269 INFO:tasks.rados.rados.0.smithi159.stdout:988: writing smithi15938319-24 from 752980 to 1382194 tid 1 2024-09-13T13:50:42.270 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:42.270+0000 7f1f37764640 -1 osd.4 64 handle_osd_map injecting map crc failure 2024-09-13T13:50:42.275 INFO:tasks.rados.rados.0.smithi159.stdout:988: writing smithi15938319-24 from 1809479 to 2352980 tid 2 2024-09-13T13:50:42.275 INFO:tasks.rados.rados.0.smithi159.stdout:989: set_chunk oid 17 target oid 2024-09-13T13:50:42.279 INFO:tasks.rados.rados.0.smithi159.stdout:989: set_chunk oid 17 offset: 437760 length: 68096 target oid 39a504f023b549728c3323fb8550abbf7ef636b56572797b89b230525c29ce3e offset: 0 snap -1 2024-09-13T13:50:42.279 INFO:tasks.rados.rados.0.smithi159.stdout:990: tier_promote oid 30 2024-09-13T13:50:42.279 INFO:tasks.rados.rados.0.smithi159.stdout:981: expect (ObjNum 82 snap 4 seq_num 82) 2024-09-13T13:50:42.296 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1956378 2024-09-13T13:50:42.313 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.313 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.322 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:42.346 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2697029 2024-09-13T13:50:42.347 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:42.345+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:42.363 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.411 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:42.411 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:42.418 INFO:tasks.rados.rados.0.smithi159.stdout:976: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:42.432 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:42.432 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:42.433 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:42.432+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:42.451 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.452 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.460 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:42.486 INFO:tasks.rados.rados.0.smithi159.stdout:979: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:42.533 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:42.532+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:42.533 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.590 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.590 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.599 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:42.614 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3048786 2024-09-13T13:50:42.617 INFO:tasks.rados.rados.0.smithi159.stdout:984: expect (ObjNum 110 snap 8 seq_num 110) 2024-09-13T13:50:42.634 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:42.632+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:42.639 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:42.638+0000 7fe857f4f640 -1 osd.7 64 handle_osd_map injecting map crc failure 2024-09-13T13:50:42.718 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.718 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.727 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:42.734 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:42.734+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:42.746 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.811 INFO:tasks.rados.rados.0.smithi159.stdout:977: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:42.834 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:42.834+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:42.845 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1845118 2024-09-13T13:50:42.850 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.850 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:42.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:42.886 INFO:tasks.rados.rados.0.smithi159.stdout:982: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-13T13:50:42.886 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.935 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:42.935+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:42.950 INFO:tasks.rados.rados.0.smithi159.stdout:980: expect (ObjNum 115 snap 8 seq_num 115) 2024-09-13T13:50:42.950 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:42.991 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:42.991 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:43.015 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:43.036 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:43.035+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:43.104 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2631872 2024-09-13T13:50:43.105 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.105 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.114 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:985: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:985: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: finishing write tid 1 to smithi15938319-31 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: finishing write tid 2 to smithi15938319-31 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: finishing write tid 3 to smithi15938319-31 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: finishing write tid 4 to smithi15938319-31 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: oid 31 updating version 0 to 86 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: oid 31 updating version 86 to 87 2024-09-13T13:50:43.129 INFO:tasks.rados.rados.0.smithi159.stdout:986: oid 31 version 87 is already newer than 85 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 87 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:986: left oid 31 (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:987: finishing write tid 1 to smithi15938319-16 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:976: done (12 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:977: done (11 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:979: done (10 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:980: done (9 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:981: done (8 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:982: done (7 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:984: done (6 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:986: done (5 left) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:991: read oid 49 snap -1 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:991: expect (ObjNum 135 snap 11 seq_num 135) 2024-09-13T13:50:43.130 INFO:tasks.rados.rados.0.smithi159.stdout:992: snap_remove snap 5 2024-09-13T13:50:43.137 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:43.136+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:43.231 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.232 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.238 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:43.237+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:43.240 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:43.250 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:43.249+0000 7f2284f10640 -1 osd.10 65 handle_osd_map injecting map crc failure 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:992: done (6 left) 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:993: read oid 21 snap 10 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:993: expect (ObjNum 110 snap 8 seq_num 110) 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:994: tier_evict oid 35 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:994: tier_evict oid 35 snap -1 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:987: finishing write tid 2 to smithi15938319-16 2024-09-13T13:50:43.260 INFO:tasks.rados.rados.0.smithi159.stdout:987: finishing write tid 3 to smithi15938319-16 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:987: finishing write tid 4 to smithi15938319-16 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:987: oid 16 updating version 0 to 100 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:987: oid 16 updating version 100 to 101 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:987: oid 16 version 101 is already newer than 99 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 101 (ObjNum 139 snap 11 seq_num 139) dirty exists 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:987: left oid 16 (ObjNum 139 snap 11 seq_num 139) 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:988: finishing write tid 1 to smithi15938319-24 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:988: finishing write tid 2 to smithi15938319-24 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:989: finishing set_chunk to oid 17 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 96 (ObjNum 127 snap 9 seq_num 127) dirty exists 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:988: finishing write tid 3 to smithi15938319-24 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:988: finishing write tid 4 to smithi15938319-24 2024-09-13T13:50:43.261 INFO:tasks.rados.rados.0.smithi159.stdout:988: oid 24 updating version 0 to 72 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:988: oid 24 updating version 72 to 73 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:988: oid 24 updating version 73 to 74 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 74 (ObjNum 140 snap 11 seq_num 140) dirty exists 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:988: left oid 24 (ObjNum 140 snap 11 seq_num 140) 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:990: got (0) Success 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 69 (ObjNum 125 snap 9 seq_num 125) dirty exists 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: oid 12 updating version 0 to 71 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: oid 12 version 71 is already newer than 69 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: oid 12 version 71 is already newer than 70 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 71 (ObjNum 137 snap 11 seq_num 137) dirty exists 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: left oid 12 (ObjNum 137 snap 11 seq_num 137) 2024-09-13T13:50:43.262 INFO:tasks.rados.rados.0.smithi159.stdout:985: done (7 left) 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:987: done (6 left) 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:988: done (5 left) 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:989: done (4 left) 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:990: done (3 left) 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:995: write oid 2 current snap is 11 2024-09-13T13:50:43.263 INFO:tasks.rados.rados.0.smithi159.stdout:995: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-13T13:50:43.266 INFO:tasks.rados.rados.0.smithi159.stdout:995: writing smithi15938319-2 from 769787 to 1474250 tid 1 2024-09-13T13:50:43.272 INFO:tasks.rados.rados.0.smithi159.stdout:995: writing smithi15938319-2 from 2267961 to 2369787 tid 2 2024-09-13T13:50:43.272 INFO:tasks.rados.rados.0.smithi159.stdout:996: write oid 30 current snap is 11 2024-09-13T13:50:43.272 INFO:tasks.rados.rados.0.smithi159.stdout:996: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-13T13:50:43.281 INFO:tasks.rados.rados.0.smithi159.stdout:996: writing smithi15938319-30 from 786594 to 1566306 tid 1 2024-09-13T13:50:43.285 INFO:tasks.rados.rados.0.smithi159.stdout:996: writing smithi15938319-30 from 2326443 to 2386594 tid 2 2024-09-13T13:50:43.285 INFO:tasks.rados.rados.0.smithi159.stdout:997: write oid 50 current snap is 11 2024-09-13T13:50:43.286 INFO:tasks.rados.rados.0.smithi159.stdout:991: expect (ObjNum 135 snap 11 seq_num 135) 2024-09-13T13:50:43.286 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:43.325 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:43.339 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:43.338+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:43.365 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.365 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:43.398 INFO:tasks.rados.rados.0.smithi159.stdout:994: got (22) Invalid argument 2024-09-13T13:50:43.398 INFO:tasks.rados.rados.0.smithi159.stdout:993: expect (ObjNum 110 snap 8 seq_num 110) 2024-09-13T13:50:43.399 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:43.439 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:43.439+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:43.502 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.502 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.510 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:43.540 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:43.539+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:43.583 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:43.582+0000 7f586a715640 -1 osd.9 65 handle_osd_map injecting map crc failure 2024-09-13T13:50:43.591 INFO:tasks.rados.rados.0.smithi159.stdout:995: finishing write tid 1 to smithi15938319-2 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: finishing write tid 2 to smithi15938319-2 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: finishing write tid 3 to smithi15938319-2 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: finishing write tid 4 to smithi15938319-2 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: oid 2 updating version 0 to 72 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: oid 2 updating version 72 to 78 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: oid 2 version 78 is already newer than 77 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 78 (ObjNum 141 snap 11 seq_num 141) dirty exists 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:995: left oid 2 (ObjNum 141 snap 11 seq_num 141) 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:996: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:996: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:43.592 INFO:tasks.rados.rados.0.smithi159.stdout:997: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-13T13:50:43.596 INFO:tasks.rados.rados.0.smithi159.stdout:997: writing smithi15938319-50 from 403401 to 858362 tid 1 2024-09-13T13:50:43.603 INFO:tasks.rados.rados.0.smithi159.stdout:997: writing smithi15938319-50 from 1584925 to 2090964 tid 2 2024-09-13T13:50:43.603 INFO:tasks.rados.rados.0.smithi159.stdout:991: done (5 left) 2024-09-13T13:50:43.603 INFO:tasks.rados.rados.0.smithi159.stdout:993: done (4 left) 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:994: done (3 left) 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:995: done (2 left) 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:998: tier_promote oid 28 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: oid 30 updating version 0 to 83 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: oid 30 updating version 83 to 84 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: oid 30 version 84 is already newer than 82 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 84 (ObjNum 142 snap 11 seq_num 142) dirty exists 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: left oid 30 (ObjNum 142 snap 11 seq_num 142) 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:997: finishing write tid 1 to smithi15938319-50 2024-09-13T13:50:43.604 INFO:tasks.rados.rados.0.smithi159.stdout:996: done (2 left) 2024-09-13T13:50:43.605 INFO:tasks.rados.rados.0.smithi159.stdout:999: read oid 19 snap -1 2024-09-13T13:50:43.605 INFO:tasks.rados.rados.0.smithi159.stdout:999: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:43.605 INFO:tasks.rados.rados.0.smithi159.stdout:1000: write oid 14 current snap is 11 2024-09-13T13:50:43.605 INFO:tasks.rados.rados.0.smithi159.stdout:1000: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-13T13:50:43.608 INFO:tasks.rados.rados.0.smithi159.stdout:1000: writing smithi15938319-14 from 420208 to 950418 tid 1 2024-09-13T13:50:43.616 INFO:tasks.rados.rados.0.smithi159.stdout:1000: writing smithi15938319-14 from 1527054 to 2293104 tid 2 2024-09-13T13:50:43.616 INFO:tasks.rados.rados.0.smithi159.stdout:1001: write (excl) oid 40 current snap is 11 2024-09-13T13:50:43.616 INFO:tasks.rados.rados.0.smithi159.stdout:1001: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-13T13:50:43.622 INFO:tasks.rados.rados.0.smithi159.stdout:1001: writing smithi15938319-40 from 437015 to 1158827 tid 1 2024-09-13T13:50:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.628 INFO:tasks.rados.rados.0.smithi159.stdout:1001: writing smithi15938319-40 from 1701889 to 2211597 tid 2 2024-09-13T13:50:43.628 INFO:tasks.rados.rados.0.smithi159.stdout:1002: read oid 31 snap -1 2024-09-13T13:50:43.628 INFO:tasks.rados.rados.0.smithi159.stdout:1002: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:50:43.628 INFO:tasks.rados.rados.0.smithi159.stdout:1003: set_chunk oid 5 target oid 2024-09-13T13:50:43.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:43.634 INFO:tasks.rados.rados.0.smithi159.stdout:1003: set_chunk oid 5 offset: 3266560 length: 390656 target oid 23a7a1079ee6a025a0c97b7245b7d401668bb74fb4fdcff0e988c4842beed3a7 offset: 0 snap 9 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:1004: tier_evict oid 45 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:1004: tier_evict oid 45 snap -1 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: finishing write tid 2 to smithi15938319-50 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:1005: read oid 23 snap -1 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:1005: expect deleted 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:1006: snap_create 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: finishing write tid 3 to smithi15938319-50 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:998: got (0) Success 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 70 (ObjNum 134 snap 11 seq_num 134) dirty exists 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: finishing write tid 4 to smithi15938319-50 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: oid 50 updating version 0 to 85 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: oid 50 version 85 is already newer than 84 2024-09-13T13:50:43.635 INFO:tasks.rados.rados.0.smithi159.stdout:997: oid 50 updating version 85 to 86 2024-09-13T13:50:43.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 86 (ObjNum 143 snap 11 seq_num 143) dirty exists 2024-09-13T13:50:43.636 INFO:tasks.rados.rados.0.smithi159.stdout:997: left oid 50 (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:43.636 INFO:tasks.rados.rados.0.smithi159.stdout:999: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:43.641 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:43.639+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:43.689 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 424373 2024-09-13T13:50:43.742 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:43.741+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:43.742 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.742 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:43.772 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:43.770+0000 7f586a715640 -1 osd.9 66 handle_osd_map injecting map crc failure 2024-09-13T13:50:43.783 INFO:tasks.rados.rados.0.smithi159.stdout:1000: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: oid 14 updating version 0 to 82 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: oid 14 updating version 82 to 83 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: oid 14 updating version 83 to 84 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 84 (ObjNum 144 snap 11 seq_num 144) dirty exists 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1000: left oid 14 (ObjNum 144 snap 11 seq_num 144) 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:1001: finishing write tid 1 to smithi15938319-40 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:997: done (9 left) 2024-09-13T13:50:43.784 INFO:tasks.rados.rados.0.smithi159.stdout:998: done (8 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:999: done (7 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1000: done (6 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1006: done (5 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1007: read oid 35 snap -1 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1007: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1003: finishing set_chunk to oid 5 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 3266560 length: 390656 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 0 (ObjNum 117 snap 8 seq_num 117) dirty exists 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1004: got (22) Invalid argument 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1008: read oid 37 snap -1 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1008: expect deleted 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1003: done (6 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1004: done (5 left) 2024-09-13T13:50:43.785 INFO:tasks.rados.rados.0.smithi159.stdout:1005: done (4 left) 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1009: write oid 12 current snap is 12 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: finishing write tid 2 to smithi15938319-40 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: finishing write tid 3 to smithi15938319-40 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: finishing write tid 4 to smithi15938319-40 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: oid 40 updating version 0 to 94 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: oid 40 updating version 94 to 96 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: oid 40 version 96 is already newer than 95 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 96 (ObjNum 145 snap 11 seq_num 145) dirty exists 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1001: left oid 40 (ObjNum 145 snap 11 seq_num 145) 2024-09-13T13:50:43.786 INFO:tasks.rados.rados.0.smithi159.stdout:1002: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:50:43.843 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:43.842+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:43.858 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:43.858 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:43.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:43.899 INFO:tasks.rados.rados.0.smithi159.stdout:1007: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T13:50:43.924 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1286714 2024-09-13T13:50:43.941 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:43.951 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:43.951+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:43.971 INFO:tasks.rados.rados.0.smithi159.stdout:1009: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-13T13:50:43.977 INFO:tasks.rados.rados.0.smithi159.stdout:1009: writing smithi15938319-12 from 453822 to 1250883 tid 1 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1009: writing smithi15938319-12 from 1760371 to 2413737 tid 2 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1001: done (4 left) 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1002: done (3 left) 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1007: done (2 left) 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1008: done (1 left) 2024-09-13T13:50:43.984 INFO:tasks.rados.rados.0.smithi159.stdout:1010: set_chunk oid 39 target oid 2024-09-13T13:50:43.985 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:43.983+0000 7f1a526ee640 -1 osd.13 66 handle_osd_map injecting map crc failure 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1010: set_chunk oid 39 offset: 375296 length: 1820672 target oid 002b1e31387002e7306234c3e132c0849205138752472d8991db207512b51a4d offset: 0 snap 7 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1011: tier_promote oid 48 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: oid 12 updating version 0 to 79 2024-09-13T13:50:44.005 INFO:tasks.rados.rados.0.smithi159.stdout:1009: oid 12 version 79 is already newer than 78 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:1009: oid 12 version 79 is already newer than 77 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 79 (ObjNum 146 snap 12 seq_num 146) dirty exists 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:1009: left oid 12 (ObjNum 146 snap 12 seq_num 146) 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:1009: done (2 left) 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:1012: write oid 38 current snap is 12 2024-09-13T13:50:44.006 INFO:tasks.rados.rados.0.smithi159.stdout:1012: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-13T13:50:44.009 INFO:tasks.rados.rados.0.smithi159.stdout:1012: writing smithi15938319-38 from 470629 to 942939 tid 1 2024-09-13T13:50:44.015 INFO:tasks.rados.rados.0.smithi159.stdout:1012: writing smithi15938319-38 from 1418853 to 1932230 tid 2 2024-09-13T13:50:44.015 INFO:tasks.rados.rados.0.smithi159.stdout:1013: read oid 36 snap 10 2024-09-13T13:50:44.015 INFO:tasks.rados.rados.0.smithi159.stdout:1013: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1014: read oid 30 snap 9 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1014: expect (ObjNum 125 snap 9 seq_num 125) 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1015: write (excl) oid 10 current snap is 12 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1010: finishing set_chunk to oid 39 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 375296 length: 1820672 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 97 snap 0 seq_num 1061423876) clean dne 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1011: got (0) Success 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 84 (ObjNum 104 snap 6 seq_num 104) dirty exists 2024-09-13T13:50:44.016 INFO:tasks.rados.rados.0.smithi159.stdout:1015: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-13T13:50:44.018 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.018 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.021 INFO:tasks.rados.rados.0.smithi159.stdout:1015: writing smithi15938319-10 from 487436 to 1034995 tid 1 2024-09-13T13:50:44.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:44.029 INFO:tasks.rados.rados.0.smithi159.stdout:1015: writing smithi15938319-10 from 1760982 to 2418017 tid 2 2024-09-13T13:50:44.029 INFO:tasks.rados.rados.0.smithi159.stdout:1010: done (5 left) 2024-09-13T13:50:44.029 INFO:tasks.rados.rados.0.smithi159.stdout:1011: done (4 left) 2024-09-13T13:50:44.029 INFO:tasks.rados.rados.0.smithi159.stdout:1016: rollback oid 27 current snap is 12 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 27 to 7 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1017: read oid 29 snap -1 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1017: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1018: delete oid 46 current snap is 12 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1012: finishing write tid 1 to smithi15938319-38 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1012: finishing write tid 2 to smithi15938319-38 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1012: finishing write tid 3 to smithi15938319-38 2024-09-13T13:50:44.030 INFO:tasks.rados.rados.0.smithi159.stdout:1014: expect (ObjNum 125 snap 9 seq_num 125) 2024-09-13T13:50:44.044 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:44.043+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:44.045 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1350325 2024-09-13T13:50:44.057 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:44.080 INFO:tasks.rados.rados.0.smithi159.stdout:1013: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:50:44.080 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 36941 2024-09-13T13:50:44.146 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:44.145+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:44.175 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.175 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.181 INFO:tasks.rados.rados.0.smithi159.stdout:1015: finishing write tid 1 to smithi15938319-10 2024-09-13T13:50:44.181 INFO:tasks.rados.rados.0.smithi159.stdout:1016: finishing rollback tid 1 to smithi15938319-27 2024-09-13T13:50:44.181 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 33 snap 0 seq_num 1515038992) dirty dne 2024-09-13T13:50:44.181 INFO:tasks.rados.rados.0.smithi159.stdout:1017: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:44.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:44.246 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:44.245+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:44.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:50:44.292 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.292 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.293 INFO:tasks.rados.rados.0.smithi159.stdout:1015: finishing write tid 2 to smithi15938319-10 2024-09-13T13:50:44.293 INFO:tasks.rados.rados.0.smithi159.stdout:1015: finishing write tid 3 to smithi15938319-10 2024-09-13T13:50:44.293 INFO:tasks.rados.rados.0.smithi159.stdout:1015: finishing write tid 4 to smithi15938319-10 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1015: oid 10 updating version 0 to 113 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1015: oid 10 version 113 is already newer than 112 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1015: oid 10 version 113 is already newer than 111 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 113 (ObjNum 148 snap 12 seq_num 148) dirty exists 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1015: left oid 10 (ObjNum 148 snap 12 seq_num 148) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1013: done (6 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1014: done (5 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1015: done (4 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1016: done (3 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1017: done (2 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1018: done (1 left) 2024-09-13T13:50:44.294 INFO:tasks.rados.rados.0.smithi159.stdout:1019: delete oid 10 current snap is 12 2024-09-13T13:50:44.296 INFO:tasks.rados.rados.0.smithi159.stdout:1019: done (1 left) 2024-09-13T13:50:44.296 INFO:tasks.rados.rados.0.smithi159.stdout:1020: read oid 15 snap -1 2024-09-13T13:50:44.296 INFO:tasks.rados.rados.0.smithi159.stdout:1020: expect (ObjNum 114 snap 8 seq_num 114) 2024-09-13T13:50:44.296 INFO:tasks.rados.rados.0.smithi159.stdout:1021: read oid 9 snap -1 2024-09-13T13:50:44.297 INFO:tasks.rados.rados.0.smithi159.stdout:1021: expect (ObjNum 136 snap 11 seq_num 136) 2024-09-13T13:50:44.297 INFO:tasks.rados.rados.0.smithi159.stdout:1022: read oid 26 snap -1 2024-09-13T13:50:44.297 INFO:tasks.rados.rados.0.smithi159.stdout:1022: expect deleted 2024-09-13T13:50:44.297 INFO:tasks.rados.rados.0.smithi159.stdout:1023: snap_remove snap 10 2024-09-13T13:50:44.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:44.346 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:44.345+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:44.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:44.474 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.474 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.474 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:44.474+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:44.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:44.548 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:44.547+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:44.584 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:44.584 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:44.607 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:44.628 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:44.629 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:44.633 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.633 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:44.649 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:44.650 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:44.650 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:44.649+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:44.671 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:44.671 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:44.709 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:44.709 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:44.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:44.748 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:44.750 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:44.749+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:44.783 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:44.784 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:44.786 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:44.786 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:44.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:44.807 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:44.808 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:44.829 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:44.851 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:44.850+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:44.851 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:44.873 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:44.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:44.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:44.914 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:44.915 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:44.915 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:44.915 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:44.915 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:44.915 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:44.920+0000 7fa9c1d9b640 0 -- 172.21.15.27:0/899307391 >> v2:172.21.15.122:6808/1255553857 conn(0x7fa9c421fb90 msgr2=0x7fa9c4221f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:44.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:44.942 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:44.943 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:44.952 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:44.951+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:44.965 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:44.965 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:45.050 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:45.050 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:45.050 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:45.050 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:45.051 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:45.053 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:45.052+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:45.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:45.154 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:45.154+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:21 4.d 4:b771c1ad:::smithi15938319-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.989668488, 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010356456999999999, 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175385+0000", 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.238e-06 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176269+0000", 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088463099999999998 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176280+0000", 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0831e-05 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176289+0000", 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176354+0000", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5102000000000001e-05 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176360+0000", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0929999999999999e-06 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180157+0000", 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003796982 2024-09-13T13:50:45.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180282+0000", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012503899999999999 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180445+0000", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016250300000000001 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184519+0000", 2024-09-13T13:50:45.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040741090000000002 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185660+0000", 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001141186 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185696+0000", 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6143000000000003e-05 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185740+0000", 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3634000000000003e-05 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:24 4.d 4:b771c1ad:::smithi15938319-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.988668482999998, 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010036734, 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9100000000000004e-07 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176387+0000", 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6060000000000001e-06 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176409+0000", 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2161e-05 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176417+0000", 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2309999999999996e-06 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176484+0000", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7453999999999995e-05 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176486+0000", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.95e-06 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182553+0000", 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060667480000000003 2024-09-13T13:50:45.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182596+0000", 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2447000000000002e-05 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186229+0000", 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036334179999999998 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186308+0000", 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8979000000000004e-05 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186420+0000", 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112249 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.180 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.266898349999998, 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:50:45.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:50:45.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:45.187 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.266830493000001, 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:50:45.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:50:45.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:50:45.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.253547568000002, 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:50:45.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:50:45.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 31.0902709, 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.200 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:45.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:50:45.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1087 4.c 4:34c00adb:::smithi15938319-12:head [write 460525~690852] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 20.804772698000001, 2024-09-13T13:50:45.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010491474000000001, 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1087 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.207 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360281+0000", 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0999999999999998e-06 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364170+0000", 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038886929999999999 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364189+0000", 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9185999999999999e-05 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364198+0000", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364278+0000", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9839999999999995e-05 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364399+0000", 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-09-13T13:50:45.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364616+0000", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021726100000000001 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.366707+0000", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002091341 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370646+0000", 2024-09-13T13:50:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039392430000000003 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370714+0000", 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8069000000000002e-05 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370771+0000", 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6292000000000002e-05 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.212 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.936855881, 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.213 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:50:45.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.217 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.936821406, 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.218 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:50:45.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.222 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.93680642, 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:50:45.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:50:45.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.936792370999999, 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.228 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:50:45.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.232 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.936770656, 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.233 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:50:45.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.936738811000001, 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:50:45.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:50:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.775042084999999, 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.243 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:50:45.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.775008934999999, 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:50:45.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:50:45.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.253 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.774978717, 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.254 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:50:45.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.774936367999999, 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:50:45.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:50:45.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:50:45.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.774902150999999, 2024-09-13T13:50:45.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:50:45.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.272 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.77486888, 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:50:45.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:50:45.278 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.372585647000001, 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.279 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:50:45.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:45.282 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:45.283 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:45.267+0000 7f2282f0c640 -1 osd.10 67 handle_osd_map injecting map crc failure 2024-09-13T13:50:45.284 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:45.254+0000 7fe857f4f640 -1 osd.7 67 handle_osd_map injecting map crc failure 2024-09-13T13:50:45.284 INFO:tasks.rados.rados.0.smithi159.stdout:1023: done (4 left) 2024-09-13T13:50:45.284 INFO:tasks.rados.rados.0.smithi159.stdout:1024: tier_promote oid 31 2024-09-13T13:50:45.284 INFO:tasks.rados.rados.0.smithi159.stdout:1020: expect (ObjNum 114 snap 8 seq_num 114) 2024-09-13T13:50:45.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:45.286 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:50:45.287 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:45.268+0000 7ffaeece8640 -1 osd.0 67 handle_osd_map injecting map crc failure 2024-09-13T13:50:45.287 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:45.283+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:45.327 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.384 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:45.383+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:45.397 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:45.397 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:45.397 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:45.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:45.421 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3438642 2024-09-13T13:50:45.430 INFO:tasks.rados.rados.0.smithi159.stdout:1021: expect (ObjNum 136 snap 11 seq_num 136) 2024-09-13T13:50:45.430 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.465 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 48279 2024-09-13T13:50:45.485 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:45.484+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:45.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:45.529 INFO:tasks.rados.rados.0.smithi159.stdout:1012: finishing write tid 4 to smithi15938319-38 2024-09-13T13:50:45.529 INFO:tasks.rados.rados.0.smithi159.stdout:1012: oid 38 updating version 0 to 59 2024-09-13T13:50:45.529 INFO:tasks.rados.rados.0.smithi159.stdout:1012: oid 38 updating version 59 to 60 2024-09-13T13:50:45.529 INFO:tasks.rados.rados.0.smithi159.stdout:1012: oid 38 updating version 60 to 61 2024-09-13T13:50:45.529 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 61 (ObjNum 147 snap 12 seq_num 147) dirty exists 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1012: left oid 38 (ObjNum 147 snap 12 seq_num 147) 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1012: done (4 left) 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1020: done (3 left) 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1021: done (2 left) 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1022: done (1 left) 2024-09-13T13:50:45.530 INFO:tasks.rados.rados.0.smithi159.stdout:1025: set_chunk oid 24 target oid 2024-09-13T13:50:45.535 INFO:tasks.rados.rados.0.smithi159.stdout:1025: set_chunk oid 24 offset: 1244160 length: 416768 target oid 704f5ca4a8361350d55a3ff6da7b792a63f051275899728658f8523598fe8329 offset: 0 snap 11 2024-09-13T13:50:45.535 INFO:tasks.rados.rados.0.smithi159.stdout:1026: write oid 21 current snap is 12 2024-09-13T13:50:45.535 INFO:tasks.rados.rados.0.smithi159.stdout:1024: got (0) Success 2024-09-13T13:50:45.535 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 87 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:50:45.535 INFO:tasks.rados.rados.0.smithi159.stdout:1026: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-13T13:50:45.541 INFO:tasks.rados.rados.0.smithi159.stdout:1026: writing smithi15938319-21 from 537857 to 1311163 tid 1 2024-09-13T13:50:45.548 INFO:tasks.rados.rados.0.smithi159.stdout:1026: writing smithi15938319-21 from 1936428 to 2457143 tid 2 2024-09-13T13:50:45.548 INFO:tasks.rados.rados.0.smithi159.stdout:1024: done (2 left) 2024-09-13T13:50:45.548 INFO:tasks.rados.rados.0.smithi159.stdout:1027: write oid 12 current snap is 12 2024-09-13T13:50:45.548 INFO:tasks.rados.rados.0.smithi159.stdout:1027: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-13T13:50:45.553 INFO:tasks.rados.rados.0.smithi159.stdout:1027: writing smithi15938319-12 from 554664 to 1003219 tid 1 2024-09-13T13:50:45.559 INFO:tasks.rados.rados.0.smithi159.stdout:1027: writing smithi15938319-12 from 1478557 to 2142930 tid 2 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1028: read oid 11 snap -1 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1028: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1029: tier_evict oid 29 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1029: tier_evict oid 29 snap -1 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1025: finishing set_chunk to oid 24 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1026: finishing write tid 1 to smithi15938319-21 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1026: finishing write tid 2 to smithi15938319-21 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1025: done (4 left) 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1030: read oid 45 snap 6 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1030: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:45.560 INFO:tasks.rados.rados.0.smithi159.stdout:1031: set_chunk oid 36 target oid 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1031: set_chunk oid 36 offset: 763392 length: 563200 target oid 1d698b1b5439ad7e6e71fb0d8addb7ef11d16006ed964067c85e445b877854d9 offset: 0 snap -1 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1032: write oid 22 current snap is 12 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: finishing write tid 3 to smithi15938319-21 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: finishing write tid 4 to smithi15938319-21 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: oid 21 updating version 0 to 61 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: oid 21 updating version 61 to 62 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: oid 21 updating version 62 to 63 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 63 (ObjNum 151 snap 12 seq_num 151) dirty exists 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1026: left oid 21 (ObjNum 151 snap 12 seq_num 151) 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1027: finishing write tid 1 to smithi15938319-12 2024-09-13T13:50:45.567 INFO:tasks.rados.rados.0.smithi159.stdout:1032: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-13T13:50:45.574 INFO:tasks.rados.rados.0.smithi159.stdout:1032: writing smithi15938319-22 from 571471 to 1211628 tid 1 2024-09-13T13:50:45.580 INFO:tasks.rados.rados.0.smithi159.stdout:1032: writing smithi15938319-22 from 1653392 to 2177776 tid 2 2024-09-13T13:50:45.580 INFO:tasks.rados.rados.0.smithi159.stdout:1026: done (6 left) 2024-09-13T13:50:45.580 INFO:tasks.rados.rados.0.smithi159.stdout:1033: read oid 14 snap -1 2024-09-13T13:50:45.580 INFO:tasks.rados.rados.0.smithi159.stdout:1033: expect (ObjNum 144 snap 11 seq_num 144) 2024-09-13T13:50:45.580 INFO:tasks.rados.rados.0.smithi159.stdout:1034: set_chunk oid 30 target oid 2024-09-13T13:50:45.586 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:45.585+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1034: set_chunk oid 30 offset: 585728 length: 1354240 target oid 3c186b8a4c582237db3f0d12fc77c33eb2c0ab8215c44be8006bdde9136930f8 offset: 0 snap 11 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1035: snap_create 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1029: got (22) Invalid argument 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1027: finishing write tid 2 to smithi15938319-12 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1027: finishing write tid 3 to smithi15938319-12 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1027: finishing write tid 4 to smithi15938319-12 2024-09-13T13:50:45.594 INFO:tasks.rados.rados.0.smithi159.stdout:1027: oid 12 updating version 0 to 82 2024-09-13T13:50:45.595 INFO:tasks.rados.rados.0.smithi159.stdout:1027: oid 12 updating version 82 to 83 2024-09-13T13:50:45.595 INFO:tasks.rados.rados.0.smithi159.stdout:1027: oid 12 updating version 83 to 84 2024-09-13T13:50:45.595 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 84 (ObjNum 152 snap 12 seq_num 152) dirty exists 2024-09-13T13:50:45.595 INFO:tasks.rados.rados.0.smithi159.stdout:1027: left oid 12 (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:50:45.595 INFO:tasks.rados.rados.0.smithi159.stdout:1028: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:45.601 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 813254 2024-09-13T13:50:45.625 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:73 4.7 4:e1bd516b:::smithi15938319-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 33.297788214000001, 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060681499999999996, 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 73 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325734+0000", 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326420+0000", 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068555600000000003 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326435+0000", 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.575e-05 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326444+0000", 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8410000000000006e-06 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326517+0000", 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2784e-05 2024-09-13T13:50:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326630+0000", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112759 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326729+0000", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9761999999999998e-05 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.328491+0000", 2024-09-13T13:50:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00176133 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331735+0000", 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003244317 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331768+0000", 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2750000000000003e-05 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331799+0000", 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1281000000000001e-05 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.708228513999998, 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:50:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:50:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.703428985000002, 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:50:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:50:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.854080947, 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:50:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.549280635999999, 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:50:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.460399391999999, 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:50:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:50:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:50:45.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1111 4.7 4:e1bd516b:::smithi15938319-19:head [write 527753~708201,stat] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 20.693259069, 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060230190000000001, 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1111 2024-09-13T13:50:45.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930262+0000", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1229999999999998e-06 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930922+0000", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065976499999999998 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930939+0000", 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7158000000000001e-05 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930948+0000", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931023+0000", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5562000000000006e-05 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931094+0000", 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0445999999999996e-05 2024-09-13T13:50:45.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931198+0000", 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010367899999999999 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.932418+0000", 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012203419999999999 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936222+0000", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038036860000000001 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936256+0000", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4477000000000002e-05 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936283+0000", 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6970000000000001e-05 2024-09-13T13:50:45.669 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.349783346999999, 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:50:45.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.677 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.343923398000001, 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:50:45.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:50:45.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:50:45.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.684 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.338978757, 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:50:45.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:50:45.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.338768178999999, 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.694 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:50:45.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.338720080000002, 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.701 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:50:45.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.255639669000001, 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.708 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.713 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.255530958000001, 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:50:45.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:50:45.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.255415677999999, 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:50:45.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(4.7 53/52 [PushReplyOp(4:e63fe140:::smithi15938319-1:head)])", 2024-09-13T13:50:45.724 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.561211389, 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064523219999999999, 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8500000000000002e-07 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062312+0000", 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8639999999999998e-06 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062335+0000", 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3156999999999999e-05 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062340+0000", 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0540000000000002e-06 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068597+0000", 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062569940000000001 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068760+0000", 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016276800000000001 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1328 4.7 4:e26a40dd:::smithi15938319-25:head [write 796665~544891] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:45.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.67401585, 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085750689999999994, 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1328 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949505+0000", 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1409999999999999e-06 2024-09-13T13:50:45.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950074+0000", 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056815299999999995 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950091+0000", 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7385999999999999e-05 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950118+0000", 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7101e-05 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950198+0000", 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9474999999999994e-05 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950204+0000", 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4659999999999996e-06 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952911+0000", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027069099999999999 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952989+0000", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8485000000000005e-05 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.953127+0000", 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746799999999999 2024-09-13T13:50:45.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957228+0000", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041009810000000001 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957981+0000", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075359999999999999 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958037+0000", 2024-09-13T13:50:45.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5349999999999997e-05 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958078+0000", 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1553e-05 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:45.735 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.3814445759999998, 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:50:45.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:50:45.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:45.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.3813582589999998, 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:45.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:50:45.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.3812978889999998, 2024-09-13T13:50:45.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:45.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:45.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:45.750 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:45.751 INFO:tasks.rados.rados.0.smithi159.stdout:1030: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:45.751 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.752 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:45.750+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:45.752 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:45.786 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1073218 2024-09-13T13:50:45.820 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.853 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:45.852+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:45.855 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:45.855 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:45.855 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:45.855 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:45.863 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1031: finishing set_chunk to oid 36 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 64 (ObjNum 133 snap 9 seq_num 133) dirty exists 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: finishing write tid 1 to smithi15938319-22 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: finishing write tid 2 to smithi15938319-22 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: finishing write tid 3 to smithi15938319-22 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: finishing write tid 4 to smithi15938319-22 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: oid 22 updating version 0 to 17 2024-09-13T13:50:45.869 INFO:tasks.rados.rados.0.smithi159.stdout:1032: oid 22 version 17 is already newer than 16 2024-09-13T13:50:45.870 INFO:tasks.rados.rados.0.smithi159.stdout:1032: oid 22 updating version 17 to 18 2024-09-13T13:50:45.870 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 18 (ObjNum 153 snap 12 seq_num 153) dirty exists 2024-09-13T13:50:45.870 INFO:tasks.rados.rados.0.smithi159.stdout:1032: left oid 22 (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:50:45.870 INFO:tasks.rados.rados.0.smithi159.stdout:1033: expect (ObjNum 144 snap 11 seq_num 144) 2024-09-13T13:50:45.870 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.930 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 6] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:50:45.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T13:50:45.957 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:45.965 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:45.965 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:45.965 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:45.965 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:45.965 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:45.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:45.974 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:45.974+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:46.001 INFO:tasks.rados.rados.0.smithi159.stdout:1034: finishing set_chunk to oid 30 2024-09-13T13:50:46.055 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:46.054+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:194 4.0 4:04ca156e:::smithi15938319-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 33.359279856000001, 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042375900000000003, 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 194 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:46.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718065+0000", 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000005e-07 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718779+0000", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071307200000000003 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718789+0000", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0902e-05 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718798+0000", 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8070000000000006e-06 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718838+0000", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9759999999999999e-05 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718890+0000", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2086000000000002e-05 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.718968+0000", 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8289999999999998e-05 2024-09-13T13:50:46.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.720182+0000", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001213492 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722239+0000", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020566180000000001 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722277+0000", 2024-09-13T13:50:46.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8420999999999997e-05 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.722303+0000", 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5579e-05 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:46.085 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.307909776000002, 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:50:46.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:50:46.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.298879343999999, 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.093 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:50:46.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:50:46.097 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.646873467999999, 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:50:46.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:50:46.102 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.643839482000001, 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:50:46.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:50:46.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1121 4.4 4:22f57a3c:::smithi15938319-42:head [read 0~3008420,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.941601+0000", 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.135744117000002, 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059204050000000001, 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1121 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.107 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941601+0000", 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941601+0000", 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941601+0000", 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5200000000000002e-07 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941606+0000", 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9289999999999997e-06 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941613+0000", 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1890000000000001e-06 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941617+0000", 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7479999999999999e-06 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942697+0000", 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0010799729999999999 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.942782+0000", 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4871999999999996e-05 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.947521+0000", 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047392420000000003 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1122 4.4 4:22f57a3c:::smithi15938319-42:head [sparse-read 2759522~128932] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.941633+0000", 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.135711703999998, 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060544309999999999, 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1122 2024-09-13T13:50:46.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941633+0000", 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941633+0000", 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941634+0000", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3700000000000001e-07 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941636+0000", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.322e-06 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941641+0000", 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2859999999999999e-06 2024-09-13T13:50:46.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.941646+0000", 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.947555+0000", 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059093619999999996 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.947609+0000", 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3647999999999999e-05 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.947688+0000", 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8464e-05 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:50:46.116 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.768007364000001, 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:50:46.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.762100532, 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:50:46.121 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:50:46.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.125 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.754665097, 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:50:46.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:50:46.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8353713550000004, 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:50:46.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:50:46.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:50:46.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.136 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8353026899999998, 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.137 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:50:46.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.143 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8352525760000002, 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:50:46.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:50:46.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:50:46.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8142952149999996, 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.150 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:50:46.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:50:46.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:50:46.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8086990119999999, 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.156 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:50:46.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8071551020000003, 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:50:46.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.8070900539999997, 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.169 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:50:46.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2001484549999999, 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:50:46.175 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:50:46.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:50:46.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1974 4.4 4:21c00763:::smithi15938319-21:head [write 537857~773306] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:45.542109+0000", 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.53523556299999997, 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052392389999999997, 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1974 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542109+0000", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542109+0000", 2024-09-13T13:50:46.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542112+0000", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3719999999999999e-06 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542884+0000", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077191599999999999 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542904+0000", 2024-09-13T13:50:46.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0030999999999998e-05 2024-09-13T13:50:46.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542930+0000", 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6466999999999999e-05 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.542989+0000", 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9132000000000002e-05 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.543072+0000", 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3251000000000006e-05 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.543199+0000", 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126997 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.545888+0000", 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026886240000000001 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.547274+0000", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001385545 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.547312+0000", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8359000000000003e-05 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.547348+0000", 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6545000000000003e-05 2024-09-13T13:50:46.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1992 4.9 e68/40 4:977d569f:::smithi15938319-22:head v 68'16, mlcod=51'14)", 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:45.575098+0000", 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.50224656899999998, 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049128949999999996, 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.575098+0000", 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.575098+0000", 2024-09-13T13:50:46.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.575100+0000", 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3740000000000001e-06 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.576579+0000", 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001479822 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.576613+0000", 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3537000000000001e-05 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.576622+0000", 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2089999999999994e-06 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.576677+0000", 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5152000000000003e-05 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.576697+0000", 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0146e-05 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.579974+0000", 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003276986 2024-09-13T13:50:46.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:45.580011+0000", 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6668999999999997e-05 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.193 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.195 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:46.194+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:46.195 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1027: done (8 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1028: done (7 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1029: done (6 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1030: done (5 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1031: done (4 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1032: done (3 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1033: done (2 left) 2024-09-13T13:50:46.259 INFO:tasks.rados.rados.0.smithi159.stdout:1034: done (1 left) 2024-09-13T13:50:46.260 INFO:tasks.rados.rados.0.smithi159.stdout:1035: done (0 left) 2024-09-13T13:50:46.260 INFO:tasks.rados.rados.0.smithi159.stdout:1036: write (excl) oid 37 current snap is 13 2024-09-13T13:50:46.260 INFO:tasks.rados.rados.0.smithi159.stdout:1036: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-13T13:50:46.265 INFO:tasks.rados.rados.0.smithi159.stdout:1036: writing smithi15938319-37 from 588278 to 1303684 tid 1 2024-09-13T13:50:46.271 INFO:tasks.rados.rados.0.smithi159.stdout:1036: writing smithi15938319-37 from 1711874 to 2379916 tid 2 2024-09-13T13:50:46.271 INFO:tasks.rados.rados.0.smithi159.stdout:1037: write (excl) oid 40 current snap is 13 2024-09-13T13:50:46.271 INFO:tasks.rados.rados.0.smithi159.stdout:1037: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-13T13:50:46.279 INFO:tasks.rados.rados.0.smithi159.stdout:1037: writing smithi15938319-40 from 605085 to 1395740 tid 1 2024-09-13T13:50:46.285 INFO:tasks.rados.rados.0.smithi159.stdout:1037: writing smithi15938319-40 from 2170356 to 2605085 tid 2 2024-09-13T13:50:46.285 INFO:tasks.rados.rados.0.smithi159.stdout:1038: write oid 6 current snap is 13 2024-09-13T13:50:46.286 INFO:tasks.rados.rados.0.smithi159.stdout:1038: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-13T13:50:46.291 INFO:tasks.rados.rados.0.smithi159.stdout:1038: writing smithi15938319-6 from 621892 to 1087796 tid 1 2024-09-13T13:50:46.295 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:46.294+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:46.299 INFO:tasks.rados.rados.0.smithi159.stdout:1038: writing smithi15938319-6 from 1712485 to 2384196 tid 2 2024-09-13T13:50:46.299 INFO:tasks.rados.rados.0.smithi159.stdout:1039: write (excl) oid 9 current snap is 13 2024-09-13T13:50:46.299 INFO:tasks.rados.rados.0.smithi159.stdout:1039: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-13T13:50:46.300 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.300 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:46.300 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:46.300 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.305 INFO:tasks.rados.rados.0.smithi159.stdout:1039: writing smithi15938319-9 from 638699 to 1179852 tid 1 2024-09-13T13:50:46.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:46.311 INFO:tasks.rados.rados.0.smithi159.stdout:1039: writing smithi15938319-9 from 1770967 to 2302689 tid 2 2024-09-13T13:50:46.311 INFO:tasks.rados.rados.0.smithi159.stdout:1040: snap_remove snap 7 2024-09-13T13:50:46.396 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:46.394+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:46.411 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.411 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:46.411 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:46.411 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:46.411 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:46.496 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:46.496+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:46.562 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.328216245999997, 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:50:46.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:50:46.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.320182099999997, 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:50:46.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:50:46.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:50:46.572 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.9244839410000001, 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:50:46.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.576 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.577 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:46.597 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:46.596+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:46.680 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:46.680 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:46.698 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:46.697+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:46.791 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.792 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:46.792 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:46.792 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:46.792 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:46.798 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:46.797+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:46.801 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:46.899 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:46.898+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:46.906 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.137485451000003, 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:50:46.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:50:46.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:50:46.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.136508253000002, 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.913 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:50:46.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.134753588000002, 2024-09-13T13:50:46.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:46.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:50:46.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:50:46.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:50:46.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.134361374999997, 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:46.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:50:46.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:46.932 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.134324048000003, 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:50:46.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:50:46.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.129525694999998, 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:50:46.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:50:46.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:46.945 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.113890470000001, 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:46.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.951 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.113784119999998, 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:50:46.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:50:46.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.105850349999997, 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:46.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:50:46.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.087587606, 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.964 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:50:46.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:50:46.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:50:46.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.087461791999999, 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:50:46.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.087335002000003, 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.977 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:50:46.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:50:46.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.983 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.006136579, 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:50:46.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:50:46.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.006035128000001, 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:50:46.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:50:46.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.996 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.444682362000002, 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.997 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:50:46.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:46.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:50:47.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:50:47.002 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.438830819, 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:50:47.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.162021183, 2024-09-13T13:50:47.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:47.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:47.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.176953368, 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:47.012 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:50:47.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:50:47.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:47.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.018 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.170932092999999, 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:47.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:50:47.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:50:47.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:47.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1552 3.1 3:bd95eec2:::8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338:head [write 0~1948672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:47.024 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.814265580000001, 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011182592999999999, 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:47.025 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1552 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090023+0000", 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:47.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.090026+0000", 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2230000000000001e-06 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092134+0000", 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021088859999999999 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092153+0000", 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8260000000000001e-05 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092162+0000", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092260+0000", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7532000000000001e-05 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092430+0000", 2024-09-13T13:50:47.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017035499999999999 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.092587+0000", 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015724700000000001 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.095894+0000", 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306944 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101100+0000", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052056560000000003 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101156+0000", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6586000000000003e-05 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.101206+0000", 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9626999999999998e-05 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.031 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.031 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:47.031 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:47.031 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:47.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:47.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:47.034 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:47.032+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:47.133 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:47.132+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:47.148 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:50:47.148 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:50:47.148 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:47.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:47.175 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:47.175 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:47.197 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:47.197 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:47.218 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:47.218 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:47.235 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:47.233+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:47.240 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:47.257 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:50:47.261 INFO:tasks.rados.rados.0.smithi159.stdout:1040: done (4 left) 2024-09-13T13:50:47.261 INFO:tasks.rados.rados.0.smithi159.stdout:1041: write oid 38 current snap is 13 2024-09-13T13:50:47.261 INFO:tasks.rados.rados.0.smithi159.stdout:1036: finishing write tid 1 to smithi15938319-37 2024-09-13T13:50:47.261 INFO:tasks.rados.rados.0.smithi159.stdout:1041: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-13T13:50:47.262 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.262 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.263 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:47.267 INFO:tasks.rados.rados.0.smithi159.stdout:1041: writing smithi15938319-38 from 655506 to 1271908 tid 1 2024-09-13T13:50:47.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:47.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T13:50:47.277 INFO:tasks.rados.rados.0.smithi159.stdout:1041: writing smithi15938319-38 from 1829449 to 2504829 tid 2 2024-09-13T13:50:47.277 INFO:tasks.rados.rados.0.smithi159.stdout:1042: write (excl) oid 33 current snap is 13 2024-09-13T13:50:47.277 INFO:tasks.rados.rados.0.smithi159.stdout:1042: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-13T13:50:47.281 INFO:tasks.rados.rados.0.smithi159.stdout:1042: writing smithi15938319-33 from 672313 to 1363964 tid 1 2024-09-13T13:50:47.288 INFO:tasks.rados.rados.0.smithi159.stdout:1042: writing smithi15938319-33 from 1887931 to 2306969 tid 2 2024-09-13T13:50:47.288 INFO:tasks.rados.rados.0.smithi159.stdout:1043: rollback oid 2 current snap is 13 2024-09-13T13:50:47.288 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 2 to 12 2024-09-13T13:50:47.288 INFO:tasks.rados.rados.0.smithi159.stdout:1036: finishing write tid 2 to smithi15938319-37 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: finishing write tid 3 to smithi15938319-37 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: finishing write tid 4 to smithi15938319-37 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: oid 37 updating version 0 to 91 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: oid 37 updating version 91 to 93 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: oid 37 version 93 is already newer than 92 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 93 (ObjNum 154 snap 13 seq_num 154) dirty exists 2024-09-13T13:50:47.289 INFO:tasks.rados.rados.0.smithi159.stdout:1036: left oid 37 (ObjNum 154 snap 13 seq_num 154) 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1037: finishing write tid 1 to smithi15938319-40 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1037: finishing write tid 2 to smithi15938319-40 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1036: done (6 left) 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1044: read oid 27 snap -1 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1044: expect deleted 2024-09-13T13:50:47.290 INFO:tasks.rados.rados.0.smithi159.stdout:1045: write (excl) oid 15 current snap is 13 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: finishing write tid 3 to smithi15938319-40 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: finishing write tid 4 to smithi15938319-40 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: oid 40 updating version 0 to 102 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: oid 40 version 102 is already newer than 101 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: oid 40 updating version 102 to 103 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 103 (ObjNum 155 snap 13 seq_num 155) dirty exists 2024-09-13T13:50:47.291 INFO:tasks.rados.rados.0.smithi159.stdout:1037: left oid 40 (ObjNum 155 snap 13 seq_num 155) 2024-09-13T13:50:47.292 INFO:tasks.rados.rados.0.smithi159.stdout:1038: finishing write tid 1 to smithi15938319-6 2024-09-13T13:50:47.292 INFO:tasks.rados.rados.0.smithi159.stdout:1045: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-13T13:50:47.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:50:47.294 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:47.294 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:47.295 INFO:tasks.rados.rados.0.smithi159.stdout:1045: writing smithi15938319-15 from 689120 to 1172373 tid 1 2024-09-13T13:50:47.302 INFO:tasks.rados.rados.0.smithi159.stdout:1045: writing smithi15938319-15 from 1946413 to 2625462 tid 2 2024-09-13T13:50:47.302 INFO:tasks.rados.rados.0.smithi159.stdout:1037: done (7 left) 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1046: read oid 5 snap -1 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1046: expect (ObjNum 117 snap 8 seq_num 117) 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1047: tier_promote oid 22 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: finishing write tid 2 to smithi15938319-6 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: finishing write tid 3 to smithi15938319-6 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: finishing write tid 4 to smithi15938319-6 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: oid 6 updating version 0 to 19 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: oid 6 updating version 19 to 20 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: oid 6 updating version 20 to 21 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 21 (ObjNum 156 snap 13 seq_num 156) dirty exists 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1038: left oid 6 (ObjNum 156 snap 13 seq_num 156) 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1039: finishing write tid 1 to smithi15938319-9 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1039: finishing write tid 2 to smithi15938319-9 2024-09-13T13:50:47.303 INFO:tasks.rados.rados.0.smithi159.stdout:1039: finishing write tid 3 to smithi15938319-9 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1039: finishing write tid 4 to smithi15938319-9 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1039: oid 9 updating version 0 to 95 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1039: oid 9 updating version 95 to 96 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1039: oid 9 updating version 96 to 97 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 97 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1039: left oid 9 (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: finishing write tid 1 to smithi15938319-38 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: finishing write tid 2 to smithi15938319-38 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: finishing write tid 3 to smithi15938319-38 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: finishing write tid 4 to smithi15938319-38 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: oid 38 updating version 0 to 71 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: oid 38 version 71 is already newer than 70 2024-09-13T13:50:47.304 INFO:tasks.rados.rados.0.smithi159.stdout:1041: oid 38 version 71 is already newer than 65 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 71 (ObjNum 158 snap 13 seq_num 158) dirty exists 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1041: left oid 38 (ObjNum 158 snap 13 seq_num 158) 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: finishing write tid 1 to smithi15938319-33 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: finishing write tid 2 to smithi15938319-33 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: finishing write tid 3 to smithi15938319-33 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: finishing write tid 4 to smithi15938319-33 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: oid 33 updating version 0 to 89 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: oid 33 version 89 is already newer than 88 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: oid 33 version 89 is already newer than 87 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 89 (ObjNum 159 snap 13 seq_num 159) dirty exists 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1042: left oid 33 (ObjNum 159 snap 13 seq_num 159) 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1043: finishing rollback tid 0 to smithi15938319-2 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1043: finishing rollback tid 1 to smithi15938319-2 2024-09-13T13:50:47.305 INFO:tasks.rados.rados.0.smithi159.stdout:1043: finishing rollback tid 2 to smithi15938319-2 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 97 (ObjNum 141 snap 11 seq_num 141) dirty exists 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1045: finishing write tid 1 to smithi15938319-15 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1038: done (8 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1039: done (7 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1041: done (6 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1042: done (5 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1043: done (4 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1044: done (3 left) 2024-09-13T13:50:47.306 INFO:tasks.rados.rados.0.smithi159.stdout:1048: set_chunk oid 42 target oid 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1048: set_chunk oid 42 offset: 168448 length: 758784 target oid 824c672b78259063e97f7c5621e1a339df3a5806d7023f372dff1c694f49c72a offset: 0 snap -1 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1049: tier_promote oid 27 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1050: rollback oid 24 current snap is 13 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1045: finishing write tid 2 to smithi15938319-15 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 24 to 11 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1045: finishing write tid 3 to smithi15938319-15 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1045: finishing write tid 4 to smithi15938319-15 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1045: oid 15 updating version 0 to 91 2024-09-13T13:50:47.311 INFO:tasks.rados.rados.0.smithi159.stdout:1045: oid 15 updating version 91 to 92 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1045: oid 15 version 92 is already newer than 90 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 92 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1045: left oid 15 (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1047: got (0) Success 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 18 (ObjNum 153 snap 12 seq_num 153) dirty exists 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1045: done (5 left) 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1047: done (4 left) 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1051: write (excl) oid 32 current snap is 13 2024-09-13T13:50:47.312 INFO:tasks.rados.rados.0.smithi159.stdout:1051: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-13T13:50:47.317 INFO:tasks.rados.rados.0.smithi159.stdout:1051: writing smithi15938319-32 from 705927 to 1264429 tid 1 2024-09-13T13:50:47.323 INFO:tasks.rados.rados.0.smithi159.stdout:1051: writing smithi15938319-32 from 2004895 to 2427602 tid 2 2024-09-13T13:50:47.324 INFO:tasks.rados.rados.0.smithi159.stdout:1052: set_chunk oid 31 target oid 2024-09-13T13:50:47.327 INFO:tasks.rados.rados.0.smithi159.stdout:1052: set_chunk oid 31 offset: 880640 length: 90624 target oid a810c123d5bc5105a94c603160ec02fb42ece4cf40ecbb844033c0a1bed28bae offset: 0 snap 12 2024-09-13T13:50:47.327 INFO:tasks.rados.rados.0.smithi159.stdout:1053: read oid 22 snap -1 2024-09-13T13:50:47.327 INFO:tasks.rados.rados.0.smithi159.stdout:1053: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:50:47.328 INFO:tasks.rados.rados.0.smithi159.stdout:1054: write oid 48 current snap is 13 2024-09-13T13:50:47.328 INFO:tasks.rados.rados.0.smithi159.stdout:1054: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-13T13:50:47.335 INFO:tasks.rados.rados.0.smithi159.stdout:1054: writing smithi15938319-48 from 722734 to 1356485 tid 1 2024-09-13T13:50:47.335 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:47.334+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:47.343 INFO:tasks.rados.rados.0.smithi159.stdout:1054: writing smithi15938319-48 from 2063377 to 2722734 tid 2 2024-09-13T13:50:47.343 INFO:tasks.rados.rados.0.smithi159.stdout:1055: read oid 37 snap -1 2024-09-13T13:50:47.343 INFO:tasks.rados.rados.0.smithi159.stdout:1055: expect (ObjNum 154 snap 13 seq_num 154) 2024-09-13T13:50:47.343 INFO:tasks.rados.rados.0.smithi159.stdout:1056: tier_promote oid 11 2024-09-13T13:50:47.343 INFO:tasks.rados.rados.0.smithi159.stdout:1048: finishing set_chunk to oid 42 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 168448 length: 758784 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 90 snap 5 seq_num 90) dirty exists 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:1048: done (9 left) 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:1057: write oid 23 current snap is 13 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:1049: got (2) No such file or directory 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:1049: got expected ENOENT (src dne) 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 84 (ObjNum 33 snap 0 seq_num 1515038992) dirty dne 2024-09-13T13:50:47.344 INFO:tasks.rados.rados.0.smithi159.stdout:1057: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-13T13:50:47.350 INFO:tasks.rados.rados.0.smithi159.stdout:1057: writing smithi15938319-23 from 739541 to 1448541 tid 1 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1057: writing smithi15938319-23 from 2121859 to 2548235 tid 2 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1049: done (9 left) 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1058: read oid 40 snap -1 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1058: expect (ObjNum 155 snap 13 seq_num 155) 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1059: tier_promote oid 3 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1050: finishing rollback tid 0 to smithi15938319-24 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1050: finishing rollback tid 1 to smithi15938319-24 2024-09-13T13:50:47.357 INFO:tasks.rados.rados.0.smithi159.stdout:1060: set_chunk oid 18 target oid 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:1060: set_chunk oid 18 offset: 398848 length: 533504 target oid 3d789f70929c62a9adf48497169ec295284a45903f3ec43e7b1a4f4d67752cf0 offset: 0 snap -1 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:1061: write oid 44 current snap is 13 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:1050: finishing rollback tid 2 to smithi15938319-24 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 109 (ObjNum 140 snap 11 seq_num 140) dirty exists 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:1046: expect (ObjNum 117 snap 8 seq_num 117) 2024-09-13T13:50:47.363 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.389 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.389 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:47.407 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:47.407 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:47.428 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:47.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:47.436 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:47.435+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:47.449 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:47.449 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:47.470 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:47.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:47.487 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:47.491 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:47.534 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:47.534 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:47.537 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:47.536+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: finishing write tid 1 to smithi15938319-32 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1052: finishing set_chunk to oid 31 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: finishing write tid 2 to smithi15938319-32 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: finishing write tid 3 to smithi15938319-32 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: finishing write tid 4 to smithi15938319-32 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: oid 32 updating version 0 to 56 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: oid 32 version 56 is already newer than 52 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:1051: oid 32 version 56 is already newer than 55 2024-09-13T13:50:47.551 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 56 (ObjNum 161 snap 13 seq_num 161) dirty exists 2024-09-13T13:50:47.552 INFO:tasks.rados.rados.0.smithi159.stdout:1051: left oid 32 (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:50:47.552 INFO:tasks.rados.rados.0.smithi159.stdout:1053: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:50:47.552 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.555 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:47.555 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:47.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:47.559+0000 7f91f1d9b640 0 -- 172.21.15.27:0/1834608516 >> v2:172.21.15.122:6808/1255553857 conn(0x7f91f421fa90 msgr2=0x7f91f4221ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:47.579 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.603 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:47.614 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.614 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:47.636 INFO:tasks.rados.rados.0.smithi159.stdout:1054: finishing write tid 1 to smithi15938319-48 2024-09-13T13:50:47.636 INFO:tasks.rados.rados.0.smithi159.stdout:1056: got (0) Success 2024-09-13T13:50:47.637 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 33 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-09-13T13:50:47.637 INFO:tasks.rados.rados.0.smithi159.stdout:1054: finishing write tid 2 to smithi15938319-48 2024-09-13T13:50:47.637 INFO:tasks.rados.rados.0.smithi159.stdout:1054: finishing write tid 3 to smithi15938319-48 2024-09-13T13:50:47.637 INFO:tasks.rados.rados.0.smithi159.stdout:1061: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-13T13:50:47.638 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:47.637+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:47.642 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:47.644 INFO:tasks.rados.rados.0.smithi159.stdout:1061: writing smithi15938319-44 from 756348 to 1540597 tid 1 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1061: writing smithi15938319-44 from 2063988 to 2750375 tid 2 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1046: done (12 left) 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1050: done (11 left) 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1051: done (10 left) 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1052: done (9 left) 2024-09-13T13:50:47.652 INFO:tasks.rados.rados.0.smithi159.stdout:1053: done (8 left) 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1056: done (7 left) 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1062: read oid 17 snap -1 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1062: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1063: snap_create 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1054: finishing write tid 4 to smithi15938319-48 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1054: oid 48 updating version 0 to 128 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1054: oid 48 version 128 is already newer than 126 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1054: oid 48 version 128 is already newer than 127 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 128 (ObjNum 162 snap 13 seq_num 162) dirty exists 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1054: left oid 48 (ObjNum 162 snap 13 seq_num 162) 2024-09-13T13:50:47.653 INFO:tasks.rados.rados.0.smithi159.stdout:1055: expect (ObjNum 154 snap 13 seq_num 154) 2024-09-13T13:50:47.700 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1154740 2024-09-13T13:50:47.715 INFO:tasks.rados.rados.0.smithi159.stdout:1057: finishing write tid 1 to smithi15938319-23 2024-09-13T13:50:47.715 INFO:tasks.rados.rados.0.smithi159.stdout:1057: finishing write tid 2 to smithi15938319-23 2024-09-13T13:50:47.715 INFO:tasks.rados.rados.0.smithi159.stdout:1057: finishing write tid 3 to smithi15938319-23 2024-09-13T13:50:47.715 INFO:tasks.rados.rados.0.smithi159.stdout:1057: finishing write tid 4 to smithi15938319-23 2024-09-13T13:50:47.715 INFO:tasks.rados.rados.0.smithi159.stdout:1057: oid 23 updating version 0 to 92 2024-09-13T13:50:47.716 INFO:tasks.rados.rados.0.smithi159.stdout:1057: oid 23 version 92 is already newer than 90 2024-09-13T13:50:47.716 INFO:tasks.rados.rados.0.smithi159.stdout:1057: oid 23 version 92 is already newer than 91 2024-09-13T13:50:47.716 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 92 (ObjNum 163 snap 13 seq_num 163) dirty exists 2024-09-13T13:50:47.716 INFO:tasks.rados.rados.0.smithi159.stdout:1057: left oid 23 (ObjNum 163 snap 13 seq_num 163) 2024-09-13T13:50:47.716 INFO:tasks.rados.rados.0.smithi159.stdout:1058: expect (ObjNum 155 snap 13 seq_num 155) 2024-09-13T13:50:47.725 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.726 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.734 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:47.770 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.778 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:47.777+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1059: got (0) Success 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 62 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1060: finishing set_chunk to oid 18 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 56 (ObjNum 123 snap 8 seq_num 123) dirty exists 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1061: finishing write tid 1 to smithi15938319-44 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1061: finishing write tid 2 to smithi15938319-44 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1061: finishing write tid 3 to smithi15938319-44 2024-09-13T13:50:47.808 INFO:tasks.rados.rados.0.smithi159.stdout:1061: finishing write tid 4 to smithi15938319-44 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:1061: oid 44 updating version 0 to 95 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:1061: oid 44 updating version 95 to 96 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:1061: oid 44 version 96 is already newer than 94 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 96 (ObjNum 164 snap 13 seq_num 164) dirty exists 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:1061: left oid 44 (ObjNum 164 snap 13 seq_num 164) 2024-09-13T13:50:47.809 INFO:tasks.rados.rados.0.smithi159.stdout:1062: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:50:47.837 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.838 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.840 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:47.839+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:47.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:47.870 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:47.941 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:47.940+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:47.988 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:47.988 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:47.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:48.042 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:48.041+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:48.099 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.099 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:48.152 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:48.151+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:48.209 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.210 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:48.244 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:48.243+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:48.265 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:50:48.266 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:48.264+0000 7f1f37764640 -1 osd.4 70 handle_osd_map injecting map crc failure 2024-09-13T13:50:48.271 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:50:48.272 INFO:tasks.rados.rados.0.smithi159.stdout:1054: done (8 left) 2024-09-13T13:50:48.272 INFO:tasks.rados.rados.0.smithi159.stdout:1055: done (7 left) 2024-09-13T13:50:48.272 INFO:tasks.rados.rados.0.smithi159.stdout:1057: done (6 left) 2024-09-13T13:50:48.272 INFO:tasks.rados.rados.0.smithi159.stdout:1058: done (5 left) 2024-09-13T13:50:48.272 INFO:tasks.rados.rados.0.smithi159.stdout:1059: done (4 left) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1060: done (3 left) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1061: done (2 left) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1062: done (1 left) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1063: done (0 left) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1064: rollback oid 44 current snap is 14 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 44 to 12 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1065: tier_flush oid 18 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1065: tier_flush oid 18 snap -1 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1066: read oid 29 snap 12 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1066: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:48.273 INFO:tasks.rados.rados.0.smithi159.stdout:1067: set_chunk oid 21 target oid 2024-09-13T13:50:48.285 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:48.283+0000 7faeaa210640 -1 osd.8 70 handle_osd_map injecting map crc failure 2024-09-13T13:50:48.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1067: set_chunk oid 21 offset: 1020416 length: 1478656 target oid 9896055a36f5da81c33bc7587eb66bf3f324fed501a8c6222db68b28f0e5897e offset: 0 snap -1 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1068: read oid 30 snap -1 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1068: expect (ObjNum 142 snap 11 seq_num 142) 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1069: set_chunk oid 27 target oid 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1069: set_chunk oid 27 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1070: delete oid 11 current snap is 14 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1065: got (16) Device or resource busy 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 18 v 56 (ObjNum 123 snap 8 seq_num 123) unflushed 13 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1064: finishing rollback tid 0 to smithi15938319-44 2024-09-13T13:50:48.291 INFO:tasks.rados.rados.0.smithi159.stdout:1066: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:50:48.292 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:48.290+0000 7f1a526ee640 -1 osd.13 70 handle_osd_map injecting map crc failure 2024-09-13T13:50:48.341 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.345 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:48.344+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:48.365 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.365 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.374 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:48.390 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:1064: finishing rollback tid 1 to smithi15938319-44 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:1064: finishing rollback tid 2 to smithi15938319-44 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 100 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:1064: done (6 left) 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:1065: done (5 left) 2024-09-13T13:50:48.439 INFO:tasks.rados.rados.0.smithi159.stdout:1066: done (4 left) 2024-09-13T13:50:48.440 INFO:tasks.rados.rados.0.smithi159.stdout:1070: done (3 left) 2024-09-13T13:50:48.440 INFO:tasks.rados.rados.0.smithi159.stdout:1071: delete oid 49 current snap is 14 2024-09-13T13:50:48.440 INFO:tasks.rados.rados.0.smithi159.stdout:1068: expect (ObjNum 142 snap 11 seq_num 142) 2024-09-13T13:50:48.446 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:48.445+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:48.475 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.517 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.518 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.526 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:48.547 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:48.546+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:48.548 INFO:tasks.rados.rados.0.smithi159.stdout:1069: finishing set_chunk to oid 27 2024-09-13T13:50:48.548 INFO:tasks.rados.rados.0.smithi159.stdout:1069: got expected ENOENT (src dne) 2024-09-13T13:50:48.548 INFO:tasks.rados.rados.0.smithi159.stdout:1067: finishing set_chunk to oid 21 2024-09-13T13:50:48.548 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 63 (ObjNum 151 snap 12 seq_num 151) dirty exists 2024-09-13T13:50:48.549 INFO:tasks.rados.rados.0.smithi159.stdout:1067: done (3 left) 2024-09-13T13:50:48.549 INFO:tasks.rados.rados.0.smithi159.stdout:1068: done (2 left) 2024-09-13T13:50:48.549 INFO:tasks.rados.rados.0.smithi159.stdout:1069: done (1 left) 2024-09-13T13:50:48.549 INFO:tasks.rados.rados.0.smithi159.stdout:1071: done (0 left) 2024-09-13T13:50:48.549 INFO:tasks.rados.rados.0.smithi159.stdout:1072: snap_create 2024-09-13T13:50:48.630 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.630 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.639 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:48.648 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:48.646+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:48.726 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:50:48.726 INFO:tasks.rados.rados.0.smithi159.stdout:1072: done (0 left) 2024-09-13T13:50:48.726 INFO:tasks.rados.rados.0.smithi159.stdout:1073: read oid 13 snap -1 2024-09-13T13:50:48.726 INFO:tasks.rados.rados.0.smithi159.stdout:1073: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:48.727 INFO:tasks.rados.rados.0.smithi159.stdout:1074: tier_promote oid 42 2024-09-13T13:50:48.727 INFO:tasks.rados.rados.0.smithi159.stdout:1075: set_chunk oid 19 target oid 2024-09-13T13:50:48.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T13:50:48.749 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:48.748+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:48.749 INFO:tasks.rados.rados.0.smithi159.stdout:1075: set_chunk oid 19 offset: 737280 length: 2065408 target oid 1b2c1e56a6eeaa0348cbbe0cf98ed5c4763fd4259f2a22e8b6e04a225d4814b5 offset: 0 snap -1 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:1076: read oid 2 snap -1 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:1076: expect (ObjNum 141 snap 11 seq_num 141) 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:1077: tier_flush oid 5 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:1077: tier_flush oid 5 snap -1 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:1073: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T13:50:48.750 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.774 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.786 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.786 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.795 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:48.799 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:1074: got (0) Success 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 31 (ObjNum 90 snap 5 seq_num 90) dirty exists 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:1075: finishing set_chunk to oid 19 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 19 set_chunk returned error code -95 offset: 737280 length: 2065408 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 0 (ObjNum 89 snap 5 seq_num 89) dirty exists 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:1077: got (16) Device or resource busy 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 5 v 49 (ObjNum 117 snap 8 seq_num 117) unflushed 9 2024-09-13T13:50:48.825 INFO:tasks.rados.rados.0.smithi159.stdout:1076: expect (ObjNum 141 snap 11 seq_num 141) 2024-09-13T13:50:48.837 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1337002 2024-09-13T13:50:48.850 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:48.849+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:48.882 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1659944 2024-09-13T13:50:48.900 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:48.901 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:48.910 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:48.951 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:48.950+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1073: done (4 left) 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1074: done (3 left) 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1075: done (2 left) 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1076: done (1 left) 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1077: done (0 left) 2024-09-13T13:50:48.952 INFO:tasks.rados.rados.0.smithi159.stdout:1078: tier_promote oid 37 2024-09-13T13:50:48.953 INFO:tasks.rados.rados.0.smithi159.stdout:1079: snap_remove snap 11 2024-09-13T13:50:49.053 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:49.052+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:49.053 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.053 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.062 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:49.154 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:49.153+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:49.165 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.165 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.174 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:49.254 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:49.253+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:49.277 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.277 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.286 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:49.356 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:49.354+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:49.391 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.391 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.400 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:49.457 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:49.456+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:49.504 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.504 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.513 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:49.559 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:49.558+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:49.615 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.615 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:49.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:49.660 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:49.658+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:49.724 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:50:49.726 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.726 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.728 INFO:tasks.rados.rados.0.smithi159.stdout:1079: done (1 left) 2024-09-13T13:50:49.728 INFO:tasks.rados.rados.0.smithi159.stdout:1080: snap_create 2024-09-13T13:50:49.728 INFO:tasks.rados.rados.0.smithi159.stdout:1078: got (0) Success 2024-09-13T13:50:49.728 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 93 (ObjNum 154 snap 13 seq_num 154) dirty exists 2024-09-13T13:50:49.736 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:49.737 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:49.736+0000 7f8f58c61640 -1 osd.6 72 handle_osd_map injecting map crc failure 2024-09-13T13:50:49.744 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T13:50:49.760 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:49.759+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:49.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:49.806 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:49.828 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:49.828 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:49.841 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.841 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.849 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:49.850 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:49.850 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:49.861 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:49.860+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:49.872 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:49.872 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:49.893 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:49.893 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:49.914 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:49.915 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:49.936 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:49.953 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:49.953 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:49.957 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:49.957 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:49.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:49.962 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:49.961+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:49.977 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:49.978 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:50.000 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:50.000 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:50.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:50.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:50.042 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:50.042 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:50.062 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:50.061+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:50.063 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:50.063 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:50.071 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.071 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:50.071 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:50.071 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:50.085 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:50.088+0000 7f67bbfff640 0 -- 172.21.15.27:0/98310195 >> v2:172.21.15.122:6808/1255553857 conn(0x7f67bc21f870 msgr2=0x7f67bc221c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:50.111 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:50.111 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:50.133 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:50.163 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:50.161+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:50.192 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:50.192 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:50.193 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:50.193 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:50.263 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:50.263+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:50.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:50:50.310 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.310 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:50.310 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:50.310 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:a1652551:::smithi15938319-33:head, version: 45'26, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@45'26, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 26.849902876000002, 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020046090000000001, 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:50.311 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458867+0000", 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.458868+0000", 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5199999999999997e-07 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.312 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459925+0000", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010574549999999999 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459958+0000", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2385999999999998e-05 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.459973+0000", 2024-09-13T13:50:50.313 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5307000000000001e-05 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460061+0000", 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7593e-05 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460077+0000", 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6286e-05 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.314 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.460872+0000", 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00079502999999999996 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 48/47 [PushOp(4:aae6a8d6:::smithi15938319-43:head, version: 42'25, data_included: [471040~777077,1757184~271499], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aae6a8d6:::smithi15938319-43:head@42'25, size: 2873964, copy_subset: [(0, 2873964)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2028683, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 26.829811696, 2024-09-13T13:50:50.315 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023915640000000001, 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:50.316 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478959+0000", 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.478973+0000", 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4603000000000001e-05 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.317 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480619+0000", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016456089999999999 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480652+0000", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3485999999999998e-05 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480672+0000", 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9434999999999999e-05 2024-09-13T13:50:50.318 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480788+0000", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116295 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.480798+0000", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0159e-05 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:23.481350+0000", 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055197700000000004 2024-09-13T13:50:50.319 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1070 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'29, mlcod=45'27)", 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.967702375999998, 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025038590000000002, 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.320 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341068+0000", 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341069+0000", 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1200000000000001e-06 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.321 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341704+0000", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063504999999999996 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341739+0000", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.451e-05 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341746+0000", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4279999999999999e-06 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.322 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341808+0000", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1482000000000005e-05 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.341825+0000", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7187e-05 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.343519+0000", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016944869999999999 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.323 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.343572+0000", 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2595e-05 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1072 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'31, mlcod=48'29)", 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.956315176, 2024-09-13T13:50:50.324 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001961282, 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.325 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352455+0000", 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.352456+0000", 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.16e-07 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.326 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353068+0000", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00061185999999999999 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353099+0000", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1364e-05 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353105+0000", 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2230000000000004e-06 2024-09-13T13:50:50.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353157+0000", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1180999999999997e-05 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353175+0000", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8136999999999999e-05 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.354365+0000", 2024-09-13T13:50:50.328 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011900859999999999 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.354416+0000", 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1615e-05 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.955356481999999, 2024-09-13T13:50:50.329 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.330 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:50:50.331 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:50:50.332 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.333 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.315904301, 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.334 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:50:50.335 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:50:50.336 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:50:50.337 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.338 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 25.313307343999998, 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:50.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:50:50.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:50:50.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.574188714000002, 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022343459999999999, 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.343 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.344 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734582+0000", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8800000000000002e-07 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734597+0000", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4083000000000001e-05 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734630+0000", 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3966e-05 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734638+0000", 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4919999999999996e-06 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734686+0000", 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8118000000000002e-05 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736816+0000", 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021298990000000002 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.573597600999999, 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:50.347 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:50:50.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.350 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.560815149, 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022820639999999999, 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747955+0000", 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747956+0000", 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.63e-07 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747966+0000", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0353999999999999e-05 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747991+0000", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4896e-05 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747997+0000", 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8730000000000004e-06 2024-09-13T13:50:50.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748883+0000", 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088666999999999999 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750237+0000", 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001353808 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.354 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.560152876, 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:50:50.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:50:50.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6652061630000006, 2024-09-13T13:50:50.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:50:50.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:50:50.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:50:50.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(3.2 64/63 [PushOp(3:626302dd:::smithi15938319-28 low_tier:head, version: 42'25, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:626302dd:::smithi15938319-28 low_tier:head@42'25, size: 2453789, copy_subset: [(0, 2453789)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643642+0000", 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6651287729999993, 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002463532, 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643642+0000", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643642+0000", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643648+0000", 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3840000000000002e-06 2024-09-13T13:50:50.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.645446+0000", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001798399 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.645569+0000", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012304199999999999 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.645586+0000", 2024-09-13T13:50:50.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6957e-05 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.645658+0000", 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1296000000000003e-05 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.645664+0000", 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8059999999999997e-06 2024-09-13T13:50:50.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646105+0000", 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00044064800000000003 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(3.2 64/63 [PushOp(3:563ca5f5:::smithi15938319-6 low_tier:head, version: 42'28, data_included: [1048576~1035459], data_size: 1035459, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:563ca5f5:::smithi15938319-6 low_tier:head@42'28, size: 2084035, copy_subset: [(0, 2084035)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2084035, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.654669+0000", 2024-09-13T13:50:50.368 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6541011040000004, 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001940624, 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.654669+0000", 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.654669+0000", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.654670+0000", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8299999999999997e-07 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.655894+0000", 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012242380000000001 2024-09-13T13:50:50.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.655921+0000", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6855999999999998e-05 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.655935+0000", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4153e-05 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.656036+0000", 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010077300000000001 2024-09-13T13:50:50.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.656044+0000", 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6940000000000001e-06 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.656610+0000", 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056542699999999997 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.372 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(3.2 64/63 [PushOp(3:755d2207:::smithi15938319-8 low_tier:head, version: 42'29, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:755d2207:::smithi15938319-8 low_tier:head@42'29, size: 2117649, copy_subset: [(0, 2117649)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.659533+0000", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6492371919999993, 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002418459, 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.659533+0000", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.659533+0000", 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.659534+0000", 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4000000000000001e-07 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661410+0000", 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0018763110000000001 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661441+0000", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0979999999999998e-05 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661452+0000", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1029999999999999e-05 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661512+0000", 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9969999999999997e-05 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661520+0000", 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9319999999999996e-06 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.661952+0000", 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00043159600000000002 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:aebdca62:::smithi15938319-23:9, version: 59'55, data_included: [442368~1015570,1560576~33006], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aebdca62:::smithi15938319-23:9@59'55, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 9=[]:{9: [9, 8, 7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1593582, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.376 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.663353+0000", 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.645417385, 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002064677, 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.663353+0000", 2024-09-13T13:50:50.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.663353+0000", 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.663354+0000", 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1500000000000004e-07 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.664592+0000", 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0012382199999999999 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.664623+0000", 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1133999999999997e-05 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.664634+0000", 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.119e-05 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.379 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.664720+0000", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6274999999999996e-05 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.664735+0000", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4436e-05 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.665418+0000", 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068280700000000001 2024-09-13T13:50:50.380 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [413472~796773,1710446~251803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.677507+0000", 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6312631169999996, 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019689690000000001, 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.677507+0000", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.677507+0000", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.677508+0000", 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9999999999999998e-07 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.678619+0000", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011115929999999999 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.678641+0000", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2069000000000002e-05 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.678651+0000", 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2490000000000007e-06 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.678746+0000", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5672000000000002e-05 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.678763+0000", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6328999999999999e-05 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.679476+0000", 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00071355699999999997 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.384 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.6214910790000001, 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.385 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1843 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'67, mlcod=0'0)", 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.879486+0000", 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.4292842530000005, 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020663249999999999, 2024-09-13T13:50:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879486+0000", 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879486+0000", 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879487+0000", 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8999999999999996e-07 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879495+0000", 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8089999999999993e-06 2024-09-13T13:50:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879516+0000", 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1297e-05 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879523+0000", 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4379999999999998e-06 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879568+0000", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4480000000000001e-05 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.879584+0000", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5804999999999999e-05 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881498+0000", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001914832 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881552+0000", 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4073999999999998e-05 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.4276774959999994, 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:50:50.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.398 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:50.401 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:50.398+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:50.500 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:50.499+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:50.548 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.548 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:50.549 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:50.549 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:50.601 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:50.601+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:50.702 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:50.701+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:50.704 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.705 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:50.705 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:50.705 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:50.705 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:50.728 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:50:50.729 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:50:50.732 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:50.731+0000 7f26bcfe2640 -1 osd.3 73 handle_osd_map injecting map crc failure 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1078: done (1 left) 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1080: done (0 left) 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1081: read oid 21 snap -1 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1081: expect (ObjNum 151 snap 12 seq_num 151) 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1082: read oid 33 snap 12 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1082: expect (ObjNum 101 snap 6 seq_num 101) 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1083: write oid 13 current snap is 16 2024-09-13T13:50:50.733 INFO:tasks.rados.rados.0.smithi159.stdout:1083: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-13T13:50:50.735 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:50.734+0000 7ffaeece8640 -1 osd.0 73 handle_osd_map injecting map crc failure 2024-09-13T13:50:50.738 INFO:tasks.rados.rados.0.smithi159.stdout:1083: writing smithi15938319-13 from 406769 to 1016765 tid 1 2024-09-13T13:50:50.744 INFO:tasks.rados.rados.0.smithi159.stdout:1083: writing smithi15938319-13 from 1439434 to 1873148 tid 2 2024-09-13T13:50:50.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T13:50:50.749 INFO:tasks.rados.rados.0.smithi159.stdout:1083: writing smithi15938319-13 from 2303522 to 2806769 tid 3 2024-09-13T13:50:50.750 INFO:tasks.rados.rados.0.smithi159.stdout:1084: set_chunk oid 5 target oid 2024-09-13T13:50:50.762 INFO:tasks.rados.rados.0.smithi159.stdout:1084: set_chunk oid 5 offset: 448512 length: 1257472 target oid 9c4dac44644c0cca716e5c113d1fde2874478f47de4dfe959c0431d196585048 offset: 0 snap -1 2024-09-13T13:50:50.762 INFO:tasks.rados.rados.0.smithi159.stdout:1085: set_chunk oid 47 target oid 2024-09-13T13:50:50.775 INFO:tasks.rados.rados.0.smithi159.stdout:1085: set_chunk oid 47 offset: 475648 length: 161280 target oid d3d87492714ae325f932c67db409c3ea1625b5c651f197f7023f63e04fc8c530 offset: 0 snap 6 2024-09-13T13:50:50.776 INFO:tasks.rados.rados.0.smithi159.stdout:1086: read oid 1 snap 14 2024-09-13T13:50:50.776 INFO:tasks.rados.rados.0.smithi159.stdout:1086: expect deleted 2024-09-13T13:50:50.776 INFO:tasks.rados.rados.0.smithi159.stdout:1087: write oid 7 current snap is 16 2024-09-13T13:50:50.776 INFO:tasks.rados.rados.0.smithi159.stdout:1087: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-13T13:50:50.779 INFO:tasks.rados.rados.0.smithi159.stdout:1087: writing smithi15938319-7 from 423576 to 825174 tid 1 2024-09-13T13:50:50.788 INFO:tasks.rados.rados.0.smithi159.stdout:1087: writing smithi15938319-7 from 1497916 to 2191641 tid 2 2024-09-13T13:50:50.788 INFO:tasks.rados.rados.0.smithi159.stdout:1088: read oid 15 snap -1 2024-09-13T13:50:50.788 INFO:tasks.rados.rados.0.smithi159.stdout:1088: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:50:50.788 INFO:tasks.rados.rados.0.smithi159.stdout:1089: write oid 49 current snap is 16 2024-09-13T13:50:50.789 INFO:tasks.rados.rados.0.smithi159.stdout:1089: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-13T13:50:50.793 INFO:tasks.rados.rados.0.smithi159.stdout:1089: writing smithi15938319-49 from 440383 to 917230 tid 1 2024-09-13T13:50:50.799 INFO:tasks.rados.rados.0.smithi159.stdout:1089: writing smithi15938319-49 from 1556398 to 1993781 tid 2 2024-09-13T13:50:50.803 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:50.802+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:50.804 INFO:tasks.rados.rados.0.smithi159.stdout:1089: writing smithi15938319-49 from 2474721 to 2840383 tid 3 2024-09-13T13:50:50.804 INFO:tasks.rados.rados.0.smithi159.stdout:1090: write oid 50 current snap is 16 2024-09-13T13:50:50.805 INFO:tasks.rados.rados.0.smithi159.stdout:1083: finishing write tid 1 to smithi15938319-13 2024-09-13T13:50:50.805 INFO:tasks.rados.rados.0.smithi159.stdout:1081: expect (ObjNum 151 snap 12 seq_num 151) 2024-09-13T13:50:50.845 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:50.845 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:50.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:50.845 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:50.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:200 4.e 4:709ad093:::smithi15938319-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.110232779999997, 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075516929999999999, 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 200 2024-09-13T13:50:50.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732826+0000", 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2300000000000002e-07 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732828+0000", 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.328e-06 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732833+0000", 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7269999999999998e-06 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732838+0000", 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2460000000000003e-06 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734037+0000", 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00119889 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734078+0000", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1675000000000003e-05 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740269+0000", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00619018 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740318+0000", 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9521999999999998e-05 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740377+0000", 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8801999999999997e-05 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:217 3.1 e41/33 3:90dfe591:::smithi15938319-18 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.059065891000003, 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097209170000000008, 2024-09-13T13:50:50.851 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783993+0000", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787823+0000", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038302620000000001 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787880+0000", 2024-09-13T13:50:50.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6681999999999997e-05 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787888+0000", 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6270000000000006e-06 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787940+0000", 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2132000000000002e-05 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787966+0000", 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5741999999999998e-05 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793657+0000", 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056903739999999998 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793713+0000", 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6449000000000001e-05 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.855 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.856 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.856 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:223 3.1 e41/33 3:932c05fb:::smithi15938319-24 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.055227657000003, 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009035576, 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787831+0000", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790933+0000", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031022459999999999 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791063+0000", 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129374 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791071+0000", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0020000000000006e-06 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791120+0000", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9348999999999998e-05 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791135+0000", 2024-09-13T13:50:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5108e-05 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796837+0000", 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057013999999999997 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796866+0000", 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9295999999999999e-05 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.861 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:228 3.1 e41/33 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'5, mlcod=0'0)", 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.045279518999997, 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011341255999999999, 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:50.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797780+0000", 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.283e-06 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800442+0000", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026622740000000001 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800472+0000", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9765999999999999e-05 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800477+0000", 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2429999999999996e-06 2024-09-13T13:50:50.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800515+0000", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7750000000000003e-05 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800533+0000", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8657000000000001e-05 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809070+0000", 2024-09-13T13:50:50.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085367659999999995 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809120+0000", 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9517e-05 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:220 3.1 e41/33 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:50.866 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.038912572000001, 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015696446999999999, 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:50:50.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804147+0000", 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1629999999999999e-06 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810603+0000", 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064567410000000002 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810700+0000", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6526000000000004e-05 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810708+0000", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9009999999999994e-06 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810749+0000", 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:50:50.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810767+0000", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7798000000000001e-05 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819801+0000", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090344989999999997 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819842+0000", 2024-09-13T13:50:50.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.053e-05 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:231 3.1 e41/33 3:9f3f9b00:::smithi15938319-32 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.032446272999998, 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012065741, 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.871 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810613+0000", 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8400000000000002e-07 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813879+0000", 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003266658 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813938+0000", 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8579000000000003e-05 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813945+0000", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3540000000000002e-06 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813991+0000", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5203000000000001e-05 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.814007+0000", 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6850999999999998e-05 2024-09-13T13:50:50.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822644+0000", 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086365629999999999 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822677+0000", 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3549e-05 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.875 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:232 3.1 e41/33 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'8, mlcod=41'4)", 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.029173630999999, 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088135090000000006, 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813885+0000", 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6899999999999997e-07 2024-09-13T13:50:50.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817413+0000", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035275020000000001 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817568+0000", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155131 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817574+0000", 2024-09-13T13:50:50.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4479999999999999e-06 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817639+0000", 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5221000000000005e-05 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817657+0000", 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7450000000000001e-05 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822683+0000", 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050262070000000004 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822698+0000", 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4881e-05 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.880 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:235 3.1 e41/33 3:9761ca94:::smithi15938319-36 low_tier:head v 41'9, mlcod=41'4)", 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.025638602999997, 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090169459999999996, 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817420+0000", 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2300000000000001e-07 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821383+0000", 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039627890000000004 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821417+0000", 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4496999999999999e-05 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821424+0000", 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0319999999999998e-06 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821470+0000", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5748000000000003e-05 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821488+0000", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8207e-05 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826398+0000", 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004909409 2024-09-13T13:50:50.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826436+0000", 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8640999999999998e-05 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:242 3.1 e41/33 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'8)", 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:50.885 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.011573808000001, 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083574370000000005, 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:50:50.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831485+0000", 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2099999999999995e-07 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834193+0000", 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027077939999999999 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834220+0000", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7018000000000001e-05 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834229+0000", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7849999999999997e-06 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834275+0000", 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5961000000000003e-05 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834296+0000", 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1401999999999999e-05 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839806+0000", 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055102500000000004 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839842+0000", 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5305999999999997e-05 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:258 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.944874669999997, 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011133136, 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.890 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:50:50.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898185+0000", 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7400000000000001e-06 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898191+0000", 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898205+0000", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4628e-05 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898213+0000", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5020000000000003e-06 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898410+0000", 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000197235 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898490+0000", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9856000000000003e-05 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.904838+0000", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063482139999999996 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.905236+0000", 2024-09-13T13:50:50.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039744999999999998 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909250+0000", 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040139110000000002 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909281+0000", 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078999999999998e-05 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909316+0000", 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5778999999999997e-05 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:248 3.1 e41/33 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'13, mlcod=41'11)", 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.935859874000002, 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012034909999999999, 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.896 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.897 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907199+0000", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4499999999999996e-07 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911325+0000", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041262089999999996 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911350+0000", 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4960000000000002e-05 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911356+0000", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4770000000000002e-06 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911406+0000", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0633000000000001e-05 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911422+0000", 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5937999999999999e-05 2024-09-13T13:50:50.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919153+0000", 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077307970000000002 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919233+0000", 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0050999999999996e-05 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:267 4.e 4:721f6cf4:::smithi15938319-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.928179374999999, 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075444659999999997, 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 267 2024-09-13T13:50:50.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914879+0000", 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914880+0000", 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3200000000000001e-06 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914888+0000", 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0039999999999999e-06 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914901+0000", 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2738999999999999e-05 2024-09-13T13:50:50.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914910+0000", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4700000000000008e-06 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914965+0000", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4695999999999999e-05 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.915028+0000", 2024-09-13T13:50:50.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2921000000000003e-05 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921586+0000", 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065584730000000004 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922280+0000", 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069410100000000003 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922365+0000", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4581999999999998e-05 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922394+0000", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8997999999999999e-05 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922423+0000", 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9162e-05 2024-09-13T13:50:50.906 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:299 4.e 4:709ad093:::smithi15938319-50:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.073619280000003, 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089831609999999999, 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.907 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 299 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769441+0000", 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5830000000000001e-06 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769453+0000", 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1756000000000001e-05 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769498+0000", 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4468999999999997e-05 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769510+0000", 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7052000000000005e-05 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769700+0000", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2683999999999996e-05 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777412+0000", 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077117100000000001 2024-09-13T13:50:50.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777907+0000", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049540900000000004 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778307+0000", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039943500000000001 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778372+0000", 2024-09-13T13:50:50.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5153000000000005e-05 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778422+0000", 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9941000000000002e-05 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:358 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.768594970000002, 2024-09-13T13:50:50.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086044410000000009, 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 358 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074464+0000", 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.915 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074474+0000", 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5670000000000001e-06 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074488+0000", 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3889e-05 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074497+0000", 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.916 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074595+0000", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7899999999999994e-05 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074675+0000", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9849000000000001e-05 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077944+0000", 2024-09-13T13:50:50.917 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003269044 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082062+0000", 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041176290000000003 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082985+0000", 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092279199999999997 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.918 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083021+0000", 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6386e-05 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083067+0000", 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6483999999999997e-05 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:807 4.e 4:717ade62:::smithi15938319-24:head [write 541192~628415,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:50.919 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 32.6622409, 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010708927, 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 807 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:50.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180820+0000", 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7879999999999998e-06 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181481+0000", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066089999999999996 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181501+0000", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0503e-05 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181511+0000", 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9329999999999996e-06 2024-09-13T13:50:50.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181582+0000", 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0916999999999999e-05 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181588+0000", 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1160000000000004e-06 2024-09-13T13:50:50.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185876+0000", 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042876900000000003 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185941+0000", 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5168999999999999e-05 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.186064+0000", 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012253599999999999 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191138+0000", 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050743660000000003 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191450+0000", 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031216700000000002 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191481+0000", 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0954000000000002e-05 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191526+0000", 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4888000000000001e-05 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:864 4.e 4:70cd1372:::smithi15938319-9:head [read 569344~1232896] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:50:50.926 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 31.697000654, 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075254359999999999, 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 864 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146057+0000", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146060+0000", 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9720000000000001e-06 2024-09-13T13:50:50.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146071+0000", 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0806e-05 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146094+0000", 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2883000000000001e-05 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146105+0000", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0668e-05 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146170+0000", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5060000000000004e-05 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.146174+0000", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5990000000000004e-06 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.152136+0000", 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0059615049999999998 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.152240+0000", 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103903 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:19.153583+0000", 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00134304 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.931 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1550 4.e 4:721f6cf4:::smithi15938319-18:6 [read 1158216~2203151,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.761055875, 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090284739999999999, 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.932 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1550 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082003+0000", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082013+0000", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1330000000000005e-06 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082033+0000", 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9996e-05 2024-09-13T13:50:50.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082048+0000", 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5593000000000001e-05 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088784+0000", 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067358349999999999 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088911+0000", 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126907 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.091031+0000", 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021196560000000001 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1552 3.1 e59/33 3:bd95eec2:::8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338:head v 59'84, mlcod=59'82)", 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:50:50.936 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.750277056, 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081256560000000002, 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092783+0000", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6929999999999999e-06 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096023+0000", 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032406370000000002 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096060+0000", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7160000000000003e-05 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096070+0000", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5540000000000003e-06 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096139+0000", 2024-09-13T13:50:50.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9385000000000001e-05 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096161+0000", 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1483000000000001e-05 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100843+0000", 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046821320000000003 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100907+0000", 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3612e-05 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1583 4.e 4:721f6cf4:::smithi15938319-18:head [read 0~3361367,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.384034099000001, 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00774357, 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.941 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1583 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.29e-07 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459029+0000", 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7890000000000002e-06 2024-09-13T13:50:50.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459033+0000", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7409999999999998e-06 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459036+0000", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2279999999999998e-06 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464333+0000", 2024-09-13T13:50:50.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052973910000000003 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464381+0000", 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7741000000000003e-05 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.466767+0000", 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023864509999999999 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.945 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1598 4.e 4:717ade62:::smithi15938319-24:head [write 699191~532079] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.327209541, 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075604720000000004, 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1598 2024-09-13T13:50:50.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515850+0000", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.449e-06 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516458+0000", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060838899999999998 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516474+0000", 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5293000000000001e-05 2024-09-13T13:50:50.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516481+0000", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516553+0000", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1365999999999994e-05 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516663+0000", 2024-09-13T13:50:50.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109738 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516884+0000", 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022128499999999999 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.519137+0000", 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022529389999999998 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523336+0000", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041994830000000004 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523368+0000", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1813000000000002e-05 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523409+0000", 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0812999999999997e-05 2024-09-13T13:50:50.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.952 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.952 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.952 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:50.952 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:50.952 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:50.953 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 553720 2024-09-13T13:50:50.953 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:50.954 INFO:tasks.rados.rados.0.smithi159.stdout:1082: expect (ObjNum 101 snap 6 seq_num 101) 2024-09-13T13:50:50.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:50.955 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:50.952+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:51.054 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:51.052+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:51.062 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.062 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:51.062 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:51.062 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:51.130 INFO:tasks.rados.rados.0.smithi159.stdout:1083: finishing write tid 2 to smithi15938319-13 2024-09-13T13:50:51.130 INFO:tasks.rados.rados.0.smithi159.stdout:1083: finishing write tid 3 to smithi15938319-13 2024-09-13T13:50:51.130 INFO:tasks.rados.rados.0.smithi159.stdout:1085: finishing set_chunk to oid 47 2024-09-13T13:50:51.130 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 475648 length: 161280 2024-09-13T13:50:51.130 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 0 (ObjNum 115 snap 8 seq_num 115) dirty exists 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1084: finishing set_chunk to oid 5 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 49 (ObjNum 117 snap 8 seq_num 117) dirty exists 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: finishing write tid 1 to smithi15938319-7 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: finishing write tid 2 to smithi15938319-7 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: finishing write tid 3 to smithi15938319-7 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: finishing write tid 4 to smithi15938319-7 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: oid 7 updating version 0 to 100 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: oid 7 version 100 is already newer than 99 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: oid 7 updating version 100 to 101 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 101 (ObjNum 168 snap 16 seq_num 168) dirty exists 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1087: left oid 7 (ObjNum 168 snap 16 seq_num 168) 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1089: finishing write tid 1 to smithi15938319-49 2024-09-13T13:50:51.131 INFO:tasks.rados.rados.0.smithi159.stdout:1088: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:50:51.155 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:51.154+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:51.185 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:51.218 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.218 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:51.218 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:51.218 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:51.218 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:51.238 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:51.255 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:51.255+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:51.290 INFO:tasks.rados.rados.0.smithi159.stdout:1089: finishing write tid 2 to smithi15938319-49 2024-09-13T13:50:51.290 INFO:tasks.rados.rados.0.smithi159.stdout:1089: finishing write tid 3 to smithi15938319-49 2024-09-13T13:50:51.290 INFO:tasks.rados.rados.0.smithi159.stdout:1090: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-13T13:50:51.295 INFO:tasks.rados.rados.0.smithi159.stdout:1090: writing smithi15938319-50 from 457190 to 1009286 tid 1 2024-09-13T13:50:51.301 INFO:tasks.rados.rados.0.smithi159.stdout:1090: writing smithi15938319-50 from 1614880 to 2195921 tid 2 2024-09-13T13:50:51.305 INFO:tasks.rados.rados.0.smithi159.stdout:1090: writing smithi15938319-50 from 2785791 to 2857190 tid 3 2024-09-13T13:50:51.305 INFO:tasks.rados.rados.0.smithi159.stdout:1081: done (9 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1082: done (8 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1084: done (7 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1085: done (6 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1086: done (5 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1087: done (4 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1088: done (3 left) 2024-09-13T13:50:51.306 INFO:tasks.rados.rados.0.smithi159.stdout:1091: snap_remove snap 15 2024-09-13T13:50:51.337 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.337 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.170097761000001, 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.338 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.339 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.340 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.341 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.342 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:50:51.343 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.170071768, 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.344 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:51.345 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.347 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:50:51.348 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:19 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.165126247000003, 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096180398, 2024-09-13T13:50:51.349 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.350 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170583+0000", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170585+0000", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1090000000000001e-06 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266590+0000", 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096005323000000004 2024-09-13T13:50:51.351 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266598+0000", 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8110000000000003e-06 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266650+0000", 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2481000000000001e-05 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.352 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266658+0000", 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266763+0000", 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105209 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:50:51.353 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.064762946000002, 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.354 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.355 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:50:51.356 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:50:51.357 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:111 4.2 e41/34 4:4d382039:::smithi15938319-28:head v 41'9, mlcod=41'5)", 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:51.358 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.891723046999999, 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.06900017, 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.359 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1600000000000001e-07 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443991+0000", 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.360 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512563+0000", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.068572151999999997 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512567+0000", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7119999999999999e-06 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512607+0000", 2024-09-13T13:50:51.361 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249000000000003e-05 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512623+0000", 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5866999999999999e-05 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512961+0000", 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033803199999999999 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.362 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512986+0000", 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5079000000000002e-05 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:135 4.a 4:5004af91:::smithi15938319-34:head [setxattr _header (56) in=7b,truncate 2554631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.821802206000001, 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034151982999999997, 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.363 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 135 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.364 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513906+0000", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513907+0000", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.99e-07 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513910+0000", 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2210000000000002e-06 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.365 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513922+0000", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1858999999999999e-05 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513929+0000", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1069999999999999e-06 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514737+0000", 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00080765899999999996 2024-09-13T13:50:51.366 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514809+0000", 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1928999999999995e-05 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.514899+0000", 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0511000000000001e-05 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.367 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515444+0000", 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00054460299999999995 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547991+0000", 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032546974999999999 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548022+0000", 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1442999999999997e-05 2024-09-13T13:50:51.368 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548058+0000", 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6077e-05 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:136 4.a 4:5004af91:::smithi15938319-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.821761211000002, 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.034245790999999998, 2024-09-13T13:50:51.369 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 136 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513947+0000", 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513948+0000", 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0200000000000002e-07 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513951+0000", 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2909999999999999e-06 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513958+0000", 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7950000000000002e-06 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.513964+0000", 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.874e-06 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.372 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515064+0000", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001099865 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515106+0000", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2398000000000003e-05 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548087+0000", 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032980559 2024-09-13T13:50:51.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548133+0000", 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6343e-05 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548193+0000", 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0064e-05 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.374 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:135 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.820019944000002, 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032374238999999999, 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.375 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515689+0000", 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8099999999999999e-07 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.376 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.515690+0000", 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7000000000000003e-07 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547939+0000", 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.032248700999999998 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547943+0000", 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6990000000000002e-06 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.377 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547981+0000", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7302999999999998e-05 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.547985+0000", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4900000000000002e-06 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.548063+0000", 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7894999999999999e-05 2024-09-13T13:50:51.378 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.768727769000002, 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.379 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.380 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.381 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.382 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:50:51.383 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.384 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.768713269000003, 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.385 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.386 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:50:51.387 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.388 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.389 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.390 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.766459828999999, 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.391 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.392 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:50:51.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.394 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.644036540999998, 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.395 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:50:51.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:50:51.397 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.398 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.399 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.400 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.644005722000003, 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.401 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:50:51.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:50:51.404 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.641847614, 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:51.406 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.407 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.408 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.409 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.410 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.432313416, 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.411 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:50:51.412 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.413 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.414 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.566335883999997, 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.415 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.416 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.417 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:50:51.419 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.420 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.552660170000003, 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.421 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.422 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.423 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:50:51.424 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1799 4.a 4:5239d073:::smithi15938319-5:head [sparse-read 3908076~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:50:51.425 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.093595347999999, 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.033674065000000003, 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1799 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.426 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242114+0000", 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1999999999999999e-06 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.427 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242122+0000", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242135+0000", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2812e-05 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242142+0000", 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0890000000000002e-06 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.428 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242204+0000", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2242999999999993e-05 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242215+0000", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1223000000000001e-05 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.255996+0000", 2024-09-13T13:50:51.429 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013781112 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.258486+0000", 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0024891660000000001 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.275534+0000", 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017048747999999999 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.430 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.275599+0000", 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4821999999999997e-05 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.275787+0000", 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018819299999999999 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1800 4.a 4:5239d073:::smithi15938319-5:head [sparse-read 0~2270735] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:51.431 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.242157+0000", 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.093551689, 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039030426, 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1800 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.432 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242157+0000", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242157+0000", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242157+0000", 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:50:51.433 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.385e-06 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242173+0000", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3073e-05 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242179+0000", 2024-09-13T13:50:51.434 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5019999999999998e-06 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242261+0000", 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242273+0000", 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1798e-05 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.435 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.258776+0000", 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016503427000000001 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.258894+0000", 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117905 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.436 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.275836+0000", 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016942096 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.275943+0000", 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000107253 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.281187+0000", 2024-09-13T13:50:51.437 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052438830000000004 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1801 4.a 4:5239d073:::smithi15938319-5:head [read 3115490~850929,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.093518477, 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.039793844000000002, 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1801 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:50:51.439 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242191+0000", 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1200000000000003e-07 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242194+0000", 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.440 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4729999999999997e-06 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242206+0000", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7450000000000001e-06 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242308+0000", 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000101611 2024-09-13T13:50:51.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.242318+0000", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0288e-05 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.258939+0000", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016620269999999999 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.259049+0000", 2024-09-13T13:50:51.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010995199999999999 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.281231+0000", 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.022182849000000001 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.281350+0000", 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118257 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.281984+0000", 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063431800000000003 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.444 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:51.446 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:51.445+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:51.547 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:51.546+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:51.553 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.553 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:51.553 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:51.553 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:51.648 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:51.646+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:51.668 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:50:51.668 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:50:51.668 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:51.730 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1091: done (3 left) 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1092: tier_promote oid 3 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: finishing write tid 1 to smithi15938319-50 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: finishing write tid 2 to smithi15938319-50 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: finishing write tid 3 to smithi15938319-50 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: finishing write tid 4 to smithi15938319-50 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: finishing write tid 5 to smithi15938319-50 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: oid 50 updating version 0 to 117 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: oid 50 updating version 117 to 119 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: oid 50 version 119 is already newer than 116 2024-09-13T13:50:51.735 INFO:tasks.rados.rados.0.smithi159.stdout:1090: oid 50 version 119 is already newer than 118 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 119 (ObjNum 170 snap 16 seq_num 170) dirty exists 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1090: left oid 50 (ObjNum 170 snap 16 seq_num 170) 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: finishing write tid 4 to smithi15938319-13 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1090: done (3 left) 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1093: write (excl) oid 16 current snap is 16 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: finishing write tid 5 to smithi15938319-13 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: oid 13 updating version 0 to 60 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: oid 13 updating version 60 to 62 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: oid 13 updating version 62 to 63 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: oid 13 version 63 is already newer than 61 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 63 (ObjNum 167 snap 16 seq_num 167) dirty exists 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1083: left oid 13 (ObjNum 167 snap 16 seq_num 167) 2024-09-13T13:50:51.736 INFO:tasks.rados.rados.0.smithi159.stdout:1089: finishing write tid 4 to smithi15938319-49 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: finishing write tid 5 to smithi15938319-49 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: oid 49 updating version 0 to 29 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: oid 49 version 29 is already newer than 27 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: oid 49 version 29 is already newer than 28 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: oid 49 version 29 is already newer than 26 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 29 (ObjNum 169 snap 16 seq_num 169) dirty exists 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1089: left oid 49 (ObjNum 169 snap 16 seq_num 169) 2024-09-13T13:50:51.737 INFO:tasks.rados.rados.0.smithi159.stdout:1093: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-13T13:50:51.740 INFO:tasks.rados.rados.0.smithi159.stdout:1093: writing smithi15938319-16 from 473997 to 1101342 tid 1 2024-09-13T13:50:51.745 INFO:tasks.rados.rados.0.smithi159.stdout:1093: writing smithi15938319-16 from 1673362 to 2114414 tid 2 2024-09-13T13:50:51.749 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:51.748+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:51.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T13:50:51.750 INFO:tasks.rados.rados.0.smithi159.stdout:1093: writing smithi15938319-16 from 2529567 to 2873997 tid 3 2024-09-13T13:50:51.750 INFO:tasks.rados.rados.0.smithi159.stdout:1083: done (3 left) 2024-09-13T13:50:51.750 INFO:tasks.rados.rados.0.smithi159.stdout:1089: done (2 left) 2024-09-13T13:50:51.751 INFO:tasks.rados.rados.0.smithi159.stdout:1094: set_chunk oid 12 target oid 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1094: set_chunk oid 12 offset: 2271744 length: 111616 target oid f266d6b1dce5591bab682da6fe47b4c6a2468ae3ea34886c7930002e1c74edd1 offset: 0 snap -1 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1095: read oid 1 snap 9 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1095: expect deleted 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1096: read oid 50 snap -1 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1096: expect (ObjNum 170 snap 16 seq_num 170) 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1097: set_chunk oid 4 target oid 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1097: set_chunk oid 4 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1098: write (excl) oid 36 current snap is 16 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1093: finishing write tid 1 to smithi15938319-16 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1093: finishing write tid 2 to smithi15938319-16 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1093: finishing write tid 3 to smithi15938319-16 2024-09-13T13:50:51.754 INFO:tasks.rados.rados.0.smithi159.stdout:1098: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-13T13:50:51.759 INFO:tasks.rados.rados.0.smithi159.stdout:1098: writing smithi15938319-36 from 490804 to 1193398 tid 1 2024-09-13T13:50:51.765 INFO:tasks.rados.rados.0.smithi159.stdout:1098: writing smithi15938319-36 from 1615491 to 2200201 tid 2 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1098: writing smithi15938319-36 from 2724284 to 2890804 tid 3 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1099: read oid 47 snap -1 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1099: expect (ObjNum 115 snap 8 seq_num 115) 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1100: tier_evict oid 15 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1100: tier_evict oid 15 snap -1 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1093: finishing write tid 4 to smithi15938319-16 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1101: write (excl) oid 49 current snap is 16 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1093: finishing write tid 5 to smithi15938319-16 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1093: oid 16 updating version 0 to 134 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1093: oid 16 updating version 134 to 137 2024-09-13T13:50:51.770 INFO:tasks.rados.rados.0.smithi159.stdout:1093: oid 16 version 137 is already newer than 136 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1093: oid 16 version 137 is already newer than 135 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 137 (ObjNum 171 snap 16 seq_num 171) dirty exists 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1093: left oid 16 (ObjNum 171 snap 16 seq_num 171) 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1092: got (0) Success 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 62 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1097: finishing set_chunk to oid 4 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1097: got expected ENOENT (src dne) 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1094: finishing set_chunk to oid 12 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 84 (ObjNum 152 snap 12 seq_num 152) dirty exists 2024-09-13T13:50:51.771 INFO:tasks.rados.rados.0.smithi159.stdout:1096: expect (ObjNum 170 snap 16 seq_num 170) 2024-09-13T13:50:51.794 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2078669 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:21 4.d 4:b771c1ad:::smithi15938319-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.800 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.623269532000002, 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010356456999999999, 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 21 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175384+0000", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.175385+0000", 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.238e-06 2024-09-13T13:50:51.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176269+0000", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088463099999999998 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176280+0000", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0831e-05 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176289+0000", 2024-09-13T13:50:51.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9660000000000002e-06 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176354+0000", 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5102000000000001e-05 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176360+0000", 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0929999999999999e-06 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180157+0000", 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003796982 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180282+0000", 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012503899999999999 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.180445+0000", 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016250300000000001 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.184519+0000", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040741090000000002 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185660+0000", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001141186 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185696+0000", 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6143000000000003e-05 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.185740+0000", 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3634000000000003e-05 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:24 4.d 4:b771c1ad:::smithi15938319-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 39.622269527, 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010036734, 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.807 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 24 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176384+0000", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9100000000000004e-07 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176387+0000", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6060000000000001e-06 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176409+0000", 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2161e-05 2024-09-13T13:50:51.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176417+0000", 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2309999999999996e-06 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176484+0000", 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7453999999999995e-05 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:51.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.176486+0000", 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.95e-06 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182553+0000", 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0060667480000000003 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.182596+0000", 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2447000000000002e-05 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186229+0000", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0036334179999999998 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186308+0000", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8979000000000004e-05 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.186420+0000", 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000112249 2024-09-13T13:50:51.813 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.900499394000001, 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.814 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:50:51.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.817 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:50:51.818 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:50:51.819 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.900431537000003, 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.820 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:51.821 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.822 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.823 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:50:51.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.825 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 38.887148611999997, 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.826 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.827 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.828 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:50:51.829 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.830 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:50:51.831 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.723871944000003, 2024-09-13T13:50:51.832 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:51.833 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:50:51.834 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.835 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.836 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:50:51.837 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1087 4.c 4:34c00adb:::smithi15938319-12:head [write 460525~690852] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.438373742, 2024-09-13T13:50:51.838 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010491474000000001, 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1087 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:51.839 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360281+0000", 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0999999999999998e-06 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.840 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364170+0000", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038886929999999999 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364189+0000", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9185999999999999e-05 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364198+0000", 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-09-13T13:50:51.841 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364278+0000", 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9839999999999995e-05 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364399+0000", 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:51.842 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364616+0000", 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021726100000000001 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.366707+0000", 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002091341 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370646+0000", 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039392430000000003 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.843 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370714+0000", 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8069000000000002e-05 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370771+0000", 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6292000000000002e-05 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.844 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570456924999998, 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:50:51.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.846 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.847 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:50:51.848 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570422449999999, 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.850 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.851 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.852 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:50:51.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.854 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570407463999999, 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.855 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.856 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.857 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:50:51.858 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570393415000002, 2024-09-13T13:50:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.860 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.861 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.862 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.863 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.864 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570371699999999, 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.865 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:50:51.866 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:50:51.867 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.868 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.869 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.570339855, 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.870 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.871 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:50:51.872 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.873 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408643129000001, 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.875 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:50:51.876 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.877 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.878 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.879 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408609979000001, 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:50:51.880 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.881 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:50:51.882 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:50:51.883 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.884 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408579760999999, 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.885 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.886 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:50:51.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:50:51.888 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.889 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.890 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408537412000001, 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.891 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:50:51.892 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.893 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.894 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.895 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.896 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408503195000002, 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.897 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.898 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:50:51.899 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:50:51.900 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.901 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:50:51.902 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.408469923999998, 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:50:51.903 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.904 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:50:51.905 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.906 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:50:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.909 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.006186691, 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:50:51.910 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.911 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:50:51.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.913 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:50:51.914 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:50:51.914 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:51.915 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2276431 2024-09-13T13:50:51.915 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:51.915 INFO:tasks.rados.rados.0.smithi159.stdout:1098: finishing write tid 1 to smithi15938319-36 2024-09-13T13:50:51.915 INFO:tasks.rados.rados.0.smithi159.stdout:1098: finishing write tid 2 to smithi15938319-36 2024-09-13T13:50:51.915 INFO:tasks.rados.rados.0.smithi159.stdout:1100: got (22) Invalid argument 2024-09-13T13:50:51.916 INFO:tasks.rados.rados.0.smithi159.stdout:1101: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-13T13:50:51.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:51.917 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:51.913+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:51.917 INFO:tasks.rados.rados.0.smithi159.stdout:1101: writing smithi15938319-49 from 507611 to 1285454 tid 1 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1101: writing smithi15938319-49 from 2073973 to 2518694 tid 2 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1092: done (9 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1093: done (8 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1094: done (7 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1095: done (6 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1096: done (5 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1097: done (4 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1100: done (3 left) 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1102: write oid 14 current snap is 16 2024-09-13T13:50:51.924 INFO:tasks.rados.rados.0.smithi159.stdout:1102: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-13T13:50:51.929 INFO:tasks.rados.rados.0.smithi159.stdout:1102: writing smithi15938319-14 from 524418 to 977510 tid 1 2024-09-13T13:50:51.936 INFO:tasks.rados.rados.0.smithi159.stdout:1102: writing smithi15938319-14 from 1732455 to 2320834 tid 2 2024-09-13T13:50:51.940 INFO:tasks.rados.rados.0.smithi159.stdout:1102: writing smithi15938319-14 from 2779130 to 2924418 tid 3 2024-09-13T13:50:51.940 INFO:tasks.rados.rados.0.smithi159.stdout:1103: read oid 50 snap 13 2024-09-13T13:50:51.940 INFO:tasks.rados.rados.0.smithi159.stdout:1103: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:51.940 INFO:tasks.rados.rados.0.smithi159.stdout:1104: snap_remove snap 14 2024-09-13T13:50:52.015 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:52.014+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:52.032 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:52.032 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:52.032 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:50:52.033 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:52.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:52.116 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:52.114+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:52.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:52.184 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:52.184 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:50:52.184 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:50:52.184 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:50:52.185 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:52.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:52.217 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:52.215+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:52.280 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:73 4.7 4:e1bd516b:::smithi15938319-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 39.970448503, 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060681499999999996, 2024-09-13T13:50:52.298 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 73 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325731+0000", 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.325734+0000", 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0199999999999999e-06 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326420+0000", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068555600000000003 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326435+0000", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.575e-05 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326444+0000", 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8410000000000006e-06 2024-09-13T13:50:52.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326517+0000", 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2784e-05 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326630+0000", 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112759 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.326729+0000", 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9761999999999998e-05 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.328491+0000", 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00176133 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331735+0000", 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003244317 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331768+0000", 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2750000000000003e-05 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.331799+0000", 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1281000000000001e-05 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 39.380888802999998, 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:50:52.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:50:52.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 39.376089274000002, 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:50:52.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:50:52.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.526741235999999, 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:50:52.318 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:50:52.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.221940924999998, 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.325 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:50:52.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:50:52.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:50:52.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.133059680999999, 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:50:52.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:50:52.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1111 4.7 4:e1bd516b:::smithi15938319-19:head [write 527753~708201,stat] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 27.365919357999999, 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0060230190000000001, 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.338 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1111 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930260+0000", 2024-09-13T13:50:52.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930262+0000", 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1229999999999998e-06 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930922+0000", 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00065976499999999998 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930939+0000", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7158000000000001e-05 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.930948+0000", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8109999999999992e-06 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931023+0000", 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5562000000000006e-05 2024-09-13T13:50:52.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931094+0000", 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0445999999999996e-05 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 10", 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.931198+0000", 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010367899999999999 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.932418+0000", 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012203419999999999 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936222+0000", 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038036860000000001 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936256+0000", 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4477000000000002e-05 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.936283+0000", 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6970000000000001e-05 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.022443635999998, 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:50:52.344 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:50:52.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:50:52.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:50:52.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.016583687000001, 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:50:52.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:50:52.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.011639045999999, 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:50:52.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:50:52.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:50:52.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:50:52.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.011428467999998, 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.368 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:50:52.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:50:52.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:52.375 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.011380369000001, 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:50:52.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:50:52.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.928299958, 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:52.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:50:52.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:50:52.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.387 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.928191247000001, 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.388 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:50:52.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.928075967000002, 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:50:52.393 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:50:52.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(4.7 53/52 [PushReplyOp(4:e63fe140:::smithi15938319-1:head)])", 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.233871678, 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064523219999999999, 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062308+0000", 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8500000000000002e-07 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062312+0000", 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8639999999999998e-06 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062335+0000", 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3156999999999999e-05 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062340+0000", 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0540000000000002e-06 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068597+0000", 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062569940000000001 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068760+0000", 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016276800000000001 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.402 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1328 4.7 4:e26a40dd:::smithi15938319-25:head [write 796665~544891] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.346676138999999, 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085750689999999994, 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1328 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949505+0000", 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1409999999999999e-06 2024-09-13T13:50:52.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950074+0000", 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056815299999999995 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950091+0000", 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7385999999999999e-05 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950118+0000", 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7101e-05 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950198+0000", 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9474999999999994e-05 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950204+0000", 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4659999999999996e-06 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952911+0000", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027069099999999999 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952989+0000", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8485000000000005e-05 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.953127+0000", 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746799999999999 2024-09-13T13:50:52.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957228+0000", 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041009810000000001 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957981+0000", 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075359999999999999 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:50:52.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958037+0000", 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5349999999999997e-05 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958078+0000", 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1553e-05 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:52.409 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.054104864999999, 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:50:52.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:50:52.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:50:52.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.054018548, 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:50:52.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:50:52.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:52.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.053958178, 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:52.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:50:52.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.424 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:50:52.425 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:50:52.425 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:50:52.426 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:52.426 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:52.425+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:52.427 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:52.427 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:52.427 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:52.428 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:52.428 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:52.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:52.429 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:52.429 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:52.429 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:52.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:52.430 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:52.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:52.431 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:52.431 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:52.450 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:52.451 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:52.472 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:52.472 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:52.493 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:52.494 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:52.515 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:52.515 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:52.526 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:52.525+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:52.533 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:52.534 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:52.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:52.536 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:52.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:52.557 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:52.557 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:52.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:52.562+0000 7f2d5f7fe640 0 -- 172.21.15.27:0/56276144 >> v2:172.21.15.122:6808/1255553857 conn(0x7f2d60221f50 msgr2=0x7f2d60224340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:52.585 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:52.585 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:52.606 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:52.606 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:52.627 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:52.626+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:52.644 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:52.644 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:52.653 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:52.729 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:52.728+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:52.743 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:50:52.745 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:52.744+0000 7f7c73c21640 -1 osd.14 75 handle_osd_map injecting map crc failure 2024-09-13T13:50:52.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:50:52.750 INFO:tasks.rados.rados.0.smithi159.stdout:1104: done (5 left) 2024-09-13T13:50:52.750 INFO:tasks.rados.rados.0.smithi159.stdout:1105: write (excl) oid 40 current snap is 16 2024-09-13T13:50:52.750 INFO:tasks.rados.rados.0.smithi159.stdout:1098: finishing write tid 3 to smithi15938319-36 2024-09-13T13:50:52.750 INFO:tasks.rados.rados.0.smithi159.stdout:1105: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-13T13:50:52.756 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:52.756 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:52.756 INFO:tasks.rados.rados.0.smithi159.stdout:1105: writing smithi15938319-40 from 541225 to 1185919 tid 1 2024-09-13T13:50:52.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T13:50:52.762 INFO:tasks.rados.rados.0.smithi159.stdout:1105: writing smithi15938319-40 from 1907290 to 2355680 tid 2 2024-09-13T13:50:52.763 INFO:tasks.rados.rados.0.smithi159.stdout:1106: write oid 1 current snap is 16 2024-09-13T13:50:52.763 INFO:tasks.rados.rados.0.smithi159.stdout:1106: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-13T13:50:52.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:52.769 INFO:tasks.rados.rados.0.smithi159.stdout:1106: writing smithi15938319-1 from 558032 to 1277975 tid 1 2024-09-13T13:50:52.776 INFO:tasks.rados.rados.0.smithi159.stdout:1106: writing smithi15938319-1 from 1849419 to 2441467 tid 2 2024-09-13T13:50:52.776 INFO:tasks.rados.rados.0.smithi159.stdout:1107: read oid 28 snap -1 2024-09-13T13:50:52.776 INFO:tasks.rados.rados.0.smithi159.stdout:1107: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:52.776 INFO:tasks.rados.rados.0.smithi159.stdout:1108: delete oid 44 current snap is 16 2024-09-13T13:50:52.776 INFO:tasks.rados.rados.0.smithi159.stdout:1098: finishing write tid 4 to smithi15938319-36 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: finishing write tid 5 to smithi15938319-36 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: oid 36 updating version 0 to 99 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: oid 36 version 99 is already newer than 98 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: oid 36 version 99 is already newer than 97 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: oid 36 version 99 is already newer than 96 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 99 (ObjNum 172 snap 16 seq_num 172) dirty exists 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1098: left oid 36 (ObjNum 172 snap 16 seq_num 172) 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:1099: expect (ObjNum 115 snap 8 seq_num 115) 2024-09-13T13:50:52.777 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 559933 2024-09-13T13:50:52.784 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:52.782+0000 7ffaeece8640 -1 osd.0 75 handle_osd_map injecting map crc failure 2024-09-13T13:50:52.829 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:52.829+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:52.833 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:52.870 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:52.870 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:52.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:52.929 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:52.929+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: finishing write tid 1 to smithi15938319-49 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: finishing write tid 2 to smithi15938319-49 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: finishing write tid 3 to smithi15938319-49 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: finishing write tid 4 to smithi15938319-49 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: oid 49 updating version 0 to 32 2024-09-13T13:50:52.941 INFO:tasks.rados.rados.0.smithi159.stdout:1101: oid 49 version 32 is already newer than 30 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1101: oid 49 version 32 is already newer than 31 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 32 (ObjNum 173 snap 16 seq_num 173) dirty exists 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1101: left oid 49 (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: finishing write tid 1 to smithi15938319-14 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: finishing write tid 2 to smithi15938319-14 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: finishing write tid 3 to smithi15938319-14 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: finishing write tid 4 to smithi15938319-14 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: finishing write tid 5 to smithi15938319-14 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: oid 14 updating version 0 to 108 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: oid 14 updating version 108 to 109 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: oid 14 updating version 109 to 111 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: oid 14 version 111 is already newer than 110 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 111 (ObjNum 174 snap 16 seq_num 174) dirty exists 2024-09-13T13:50:52.942 INFO:tasks.rados.rados.0.smithi159.stdout:1102: left oid 14 (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:50:52.943 INFO:tasks.rados.rados.0.smithi159.stdout:1103: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:52.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:53.030 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:53.029+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: finishing write tid 1 to smithi15938319-40 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: finishing write tid 2 to smithi15938319-40 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: finishing write tid 3 to smithi15938319-40 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: finishing write tid 4 to smithi15938319-40 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: oid 40 updating version 0 to 121 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: oid 40 version 121 is already newer than 119 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: oid 40 version 121 is already newer than 120 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 121 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1105: left oid 40 (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:50:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1106: finishing write tid 1 to smithi15938319-1 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: finishing write tid 2 to smithi15938319-1 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: finishing write tid 3 to smithi15938319-1 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: finishing write tid 4 to smithi15938319-1 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: oid 1 updating version 0 to 86 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: oid 1 version 86 is already newer than 85 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: oid 1 version 86 is already newer than 84 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 86 (ObjNum 176 snap 16 seq_num 176) dirty exists 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1106: left oid 1 (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:50:53.043 INFO:tasks.rados.rados.0.smithi159.stdout:1107: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:53.092 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.092 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:53.131 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:53.130+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:53.204 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.204 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:50:53.232 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:53.232+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1098: done (8 left) 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1099: done (7 left) 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1101: done (6 left) 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1102: done (5 left) 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1103: done (4 left) 2024-09-13T13:50:53.235 INFO:tasks.rados.rados.0.smithi159.stdout:1105: done (3 left) 2024-09-13T13:50:53.236 INFO:tasks.rados.rados.0.smithi159.stdout:1106: done (2 left) 2024-09-13T13:50:53.236 INFO:tasks.rados.rados.0.smithi159.stdout:1107: done (1 left) 2024-09-13T13:50:53.236 INFO:tasks.rados.rados.0.smithi159.stdout:1108: done (0 left) 2024-09-13T13:50:53.236 INFO:tasks.rados.rados.0.smithi159.stdout:1109: set_chunk oid 29 target oid 2024-09-13T13:50:53.241 INFO:tasks.rados.rados.0.smithi159.stdout:1109: set_chunk oid 29 offset: 2044416 length: 20992 target oid dfc370a1851762427c8e1f4355fc87212dd169a1508c10ad0eb8ed124afa7400 offset: 0 snap -1 2024-09-13T13:50:53.241 INFO:tasks.rados.rados.0.smithi159.stdout:1110: write oid 44 current snap is 16 2024-09-13T13:50:53.241 INFO:tasks.rados.rados.0.smithi159.stdout:1110: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-13T13:50:53.246 INFO:tasks.rados.rados.0.smithi159.stdout:1110: writing smithi15938319-44 from 591646 to 1062087 tid 1 2024-09-13T13:50:53.252 INFO:tasks.rados.rados.0.smithi159.stdout:1110: writing smithi15938319-44 from 1566383 to 2162100 tid 2 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1110: writing smithi15938319-44 from 2888822 to 2991646 tid 3 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1111: read oid 2 snap -1 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1111: expect (ObjNum 141 snap 11 seq_num 141) 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1112: snap_create 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1109: finishing set_chunk to oid 29 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 57 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:50:53.257 INFO:tasks.rados.rados.0.smithi159.stdout:1110: finishing write tid 1 to smithi15938319-44 2024-09-13T13:50:53.258 INFO:tasks.rados.rados.0.smithi159.stdout:1110: finishing write tid 2 to smithi15938319-44 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: finishing write tid 3 to smithi15938319-44 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: finishing write tid 4 to smithi15938319-44 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: finishing write tid 5 to smithi15938319-44 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: oid 44 updating version 0 to 111 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: oid 44 version 111 is already newer than 109 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: oid 44 version 111 is already newer than 110 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: oid 44 updating version 111 to 112 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 112 (ObjNum 178 snap 16 seq_num 178) dirty exists 2024-09-13T13:50:53.260 INFO:tasks.rados.rados.0.smithi159.stdout:1110: left oid 44 (ObjNum 178 snap 16 seq_num 178) 2024-09-13T13:50:53.265 INFO:tasks.rados.rados.0.smithi159.stdout:1111: expect (ObjNum 141 snap 11 seq_num 141) 2024-09-13T13:50:53.267 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 443804 2024-09-13T13:50:53.315 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.316 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:50:53.333 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:53.333+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:53.426 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.426 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.434 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:53.433+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:53.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:50:53.535 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:53.534+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:53.552 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.553 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:50:53.635 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:53.635+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:53.662 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.662 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:50:53.723 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:50:53.723 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T13:50:53.726 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:53.725+0000 7faeac214640 -1 osd.8 76 handle_osd_map injecting map crc failure 2024-09-13T13:50:53.729 INFO:tasks.rados.rados.0.smithi159.stdout:1109: done (3 left) 2024-09-13T13:50:53.729 INFO:tasks.rados.rados.0.smithi159.stdout:1110: done (2 left) 2024-09-13T13:50:53.730 INFO:tasks.rados.rados.0.smithi159.stdout:1111: done (1 left) 2024-09-13T13:50:53.730 INFO:tasks.rados.rados.0.smithi159.stdout:1112: done (0 left) 2024-09-13T13:50:53.730 INFO:tasks.rados.rados.0.smithi159.stdout:1113: delete oid 50 current snap is 17 2024-09-13T13:50:53.740 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:53.740+0000 7f2284f10640 -1 osd.10 76 handle_osd_map injecting map crc failure 2024-09-13T13:50:53.741 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:53.740+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1113: done (0 left) 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1114: set_chunk oid 27 target oid 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1114: set_chunk oid 27 offset: 950272 length: 2325504 target oid smithi15938319- offset: 950272 snap 12 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1115: read oid 15 snap -1 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1115: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:50:53.743 INFO:tasks.rados.rados.0.smithi159.stdout:1116: set_chunk oid 3 target oid 2024-09-13T13:50:53.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T13:50:53.765 INFO:tasks.rados.rados.0.smithi159.stdout:1116: set_chunk oid 3 offset: 369152 length: 111616 target oid e6b4f9f8265bfe5a786cb2bd77c53a79b3e484041c7c0186a813c80a2460efe6 offset: 0 snap -1 2024-09-13T13:50:53.765 INFO:tasks.rados.rados.0.smithi159.stdout:1117: read oid 11 snap 9 2024-09-13T13:50:53.765 INFO:tasks.rados.rados.0.smithi159.stdout:1117: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:53.765 INFO:tasks.rados.rados.0.smithi159.stdout:1118: write (excl) oid 35 current snap is 17 2024-09-13T13:50:53.766 INFO:tasks.rados.rados.0.smithi159.stdout:1114: finishing set_chunk to oid 27 2024-09-13T13:50:53.766 INFO:tasks.rados.rados.0.smithi159.stdout:1114: got expected ENOENT (src dne) 2024-09-13T13:50:53.766 INFO:tasks.rados.rados.0.smithi159.stdout:1115: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:50:53.819 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.819 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:50:53.833 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2316287 2024-09-13T13:50:53.836 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:53.836+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:53.885 INFO:tasks.rados.rados.0.smithi159.stdout:1117: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:50:53.923 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:53.937 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:53.936+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:53.969 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 265675 2024-09-13T13:50:53.972 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:53.972 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:53.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:50:54.013 INFO:tasks.rados.rados.0.smithi159.stdout:1116: finishing set_chunk to oid 3 2024-09-13T13:50:54.013 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 62 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:50:54.014 INFO:tasks.rados.rados.0.smithi159.stdout:1118: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-13T13:50:54.020 INFO:tasks.rados.rados.0.smithi159.stdout:1118: writing smithi15938319-35 from 625260 to 1246199 tid 1 2024-09-13T13:50:54.025 INFO:tasks.rados.rados.0.smithi159.stdout:1118: writing smithi15938319-35 from 1683347 to 2282733 tid 2 2024-09-13T13:50:54.029 INFO:tasks.rados.rados.0.smithi159.stdout:1118: writing smithi15938319-35 from 2943668 to 3025260 tid 3 2024-09-13T13:50:54.029 INFO:tasks.rados.rados.0.smithi159.stdout:1114: done (4 left) 2024-09-13T13:50:54.029 INFO:tasks.rados.rados.0.smithi159.stdout:1115: done (3 left) 2024-09-13T13:50:54.029 INFO:tasks.rados.rados.0.smithi159.stdout:1116: done (2 left) 2024-09-13T13:50:54.030 INFO:tasks.rados.rados.0.smithi159.stdout:1117: done (1 left) 2024-09-13T13:50:54.030 INFO:tasks.rados.rados.0.smithi159.stdout:1119: write (excl) oid 2 current snap is 17 2024-09-13T13:50:54.030 INFO:tasks.rados.rados.0.smithi159.stdout:1119: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-13T13:50:54.036 INFO:tasks.rados.rados.0.smithi159.stdout:1119: writing smithi15938319-2 from 642067 to 1338255 tid 1 2024-09-13T13:50:54.042 INFO:tasks.rados.rados.0.smithi159.stdout:1119: writing smithi15938319-2 from 2025476 to 2484873 tid 2 2024-09-13T13:50:54.045 INFO:tasks.rados.rados.0.smithi159.stdout:1119: writing smithi15938319-2 from 2971091 to 3042067 tid 3 2024-09-13T13:50:54.045 INFO:tasks.rados.rados.0.smithi159.stdout:1120: read oid 8 snap -1 2024-09-13T13:50:54.045 INFO:tasks.rados.rados.0.smithi159.stdout:1120: expect deleted 2024-09-13T13:50:54.045 INFO:tasks.rados.rados.0.smithi159.stdout:1121: read oid 28 snap -1 2024-09-13T13:50:54.045 INFO:tasks.rados.rados.0.smithi159.stdout:1121: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:54.046 INFO:tasks.rados.rados.0.smithi159.stdout:1122: snap_remove snap 12 2024-09-13T13:50:54.065 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:54.065+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:54.124 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.124 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:50:54.174 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:54.173+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:54.240 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:54.239+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:54.277 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.277 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:50:54.341 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:54.341+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:54.429 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.429 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:50:54.442 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:54.441+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:54.540 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.541 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.543 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:50:54.542+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:50:54.549 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:50:54.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:54.662 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:54.661+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:54.692 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.692 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.701 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:50:54.725 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T13:50:54.728 INFO:tasks.rados.rados.0.smithi159.stdout:1122: done (4 left) 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1123: read oid 18 snap -1 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1123: expect (ObjNum 123 snap 8 seq_num 123) 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1124: tier_evict oid 34 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1124: tier_evict oid 34 snap -1 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1125: read oid 14 snap -1 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1125: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1126: read oid 11 snap -1 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1126: expect deleted 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1127: read oid 20 snap -1 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1127: expect (ObjNum 131 snap 9 seq_num 131) 2024-09-13T13:50:54.729 INFO:tasks.rados.rados.0.smithi159.stdout:1128: set_chunk oid 42 target oid 2024-09-13T13:50:54.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T13:50:54.746 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:54.746+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:54.749 INFO:tasks.rados.rados.0.smithi159.stdout:1128: set_chunk oid 42 offset: 1531392 length: 662528 target oid 307fcee7d69536e2974c75fe7730c842cef3de277bde9d7c784e4bb63c6c399e offset: 0 snap -1 2024-09-13T13:50:54.749 INFO:tasks.rados.rados.0.smithi159.stdout:1129: rollback oid 43 current snap is 17 2024-09-13T13:50:54.749 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 43 to 9 2024-09-13T13:50:54.749 INFO:tasks.rados.rados.0.smithi159.stdout:1124: got (22) Invalid argument 2024-09-13T13:50:54.749 INFO:tasks.rados.rados.0.smithi159.stdout:1127: expect (ObjNum 131 snap 9 seq_num 131) 2024-09-13T13:50:54.765 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:54.766 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:54.791 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:54.812 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:54.813 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.813 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.822 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:50:54.823 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1000427 2024-09-13T13:50:54.834 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:54.834 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:54.845 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:54.844+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:54.849 INFO:tasks.rados.rados.0.smithi159.stdout:1123: expect (ObjNum 123 snap 8 seq_num 123) 2024-09-13T13:50:54.854 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:54.854 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:54.877 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:54.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:54.900 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:54.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:54.921 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:54.922 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:54.941 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:54.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:54.945 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:54.945+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:1128: finishing set_chunk to oid 42 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 1531392 length: 662528 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 0 (ObjNum 90 snap 5 seq_num 90) dirty exists 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:1118: finishing write tid 1 to smithi15938319-35 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:1129: finishing rollback tid 0 to smithi15938319-43 2024-09-13T13:50:54.955 INFO:tasks.rados.rados.0.smithi159.stdout:1118: finishing write tid 2 to smithi15938319-35 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: finishing write tid 3 to smithi15938319-35 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: finishing write tid 4 to smithi15938319-35 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: finishing write tid 5 to smithi15938319-35 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: oid 35 updating version 0 to 143 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: oid 35 updating version 143 to 144 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: oid 35 updating version 144 to 145 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: oid 35 version 145 is already newer than 142 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 145 (ObjNum 180 snap 17 seq_num 180) dirty exists 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1118: left oid 35 (ObjNum 180 snap 17 seq_num 180) 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1129: finishing rollback tid 1 to smithi15938319-43 2024-09-13T13:50:54.956 INFO:tasks.rados.rados.0.smithi159.stdout:1129: finishing rollback tid 2 to smithi15938319-43 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 105 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1118: done (10 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1120: done (9 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1123: done (8 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1124: done (7 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1126: done (6 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1127: done (5 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1128: done (4 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1129: done (3 left) 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1130: tier_evict oid 23 2024-09-13T13:50:54.957 INFO:tasks.rados.rados.0.smithi159.stdout:1130: tier_evict oid 23 snap -1 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1131: read oid 45 snap -1 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1131: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1132: read oid 4 snap -1 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1132: expect deleted 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1133: read oid 12 snap -1 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1133: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1134: read oid 31 snap 16 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1134: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1135: write oid 6 current snap is 17 2024-09-13T13:50:54.958 INFO:tasks.rados.rados.0.smithi159.stdout:1135: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-13T13:50:54.961 INFO:tasks.rados.rados.0.smithi159.stdout:1135: writing smithi15938319-6 from 658874 to 1430311 tid 1 2024-09-13T13:50:54.963 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:54.965 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:54.965 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:54.968 INFO:tasks.rados.rados.0.smithi159.stdout:1135: writing smithi15938319-6 from 2083958 to 2687013 tid 2 2024-09-13T13:50:54.968 INFO:tasks.rados.rados.0.smithi159.stdout:1136: set_chunk oid 34 target oid 2024-09-13T13:50:54.975 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1136: set_chunk oid 34 offset: 2549760 length: 365056 target oid 7635debe9c7a303f6697f6467c52e5e18dbb2854a8825b1fc19520608792560e offset: 0 snap -1 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1137: rollback oid 43 current snap is 17 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 43 to 16 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1130: got (22) Invalid argument 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1130: done (10 left) 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1132: done (9 left) 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1138: write (excl) oid 20 current snap is 17 2024-09-13T13:50:54.976 INFO:tasks.rados.rados.0.smithi159.stdout:1138: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-13T13:50:54.982 INFO:tasks.rados.rados.0.smithi159.stdout:1138: writing smithi15938319-20 from 675681 to 1238720 tid 1 2024-09-13T13:50:54.987 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:54.989 INFO:tasks.rados.rados.0.smithi159.stdout:1138: writing smithi15938319-20 from 1858793 to 2605506 tid 2 2024-09-13T13:50:54.992 INFO:tasks.rados.rados.0.smithi159.stdout:1138: writing smithi15938319-20 from 3025937 to 3075681 tid 3 2024-09-13T13:50:54.992 INFO:tasks.rados.rados.0.smithi159.stdout:1139: read oid 21 snap -1 2024-09-13T13:50:54.992 INFO:tasks.rados.rados.0.smithi159.stdout:1139: expect (ObjNum 151 snap 12 seq_num 151) 2024-09-13T13:50:54.992 INFO:tasks.rados.rados.0.smithi159.stdout:1140: write oid 36 current snap is 17 2024-09-13T13:50:54.993 INFO:tasks.rados.rados.0.smithi159.stdout:1140: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-13T13:50:54.999 INFO:tasks.rados.rados.0.smithi159.stdout:1140: writing smithi15938319-36 from 692488 to 1330776 tid 1 2024-09-13T13:50:55.006 INFO:tasks.rados.rados.0.smithi159.stdout:1140: writing smithi15938319-36 from 1917275 to 2523999 tid 2 2024-09-13T13:50:55.006 INFO:tasks.rados.rados.0.smithi159.stdout:1141: read oid 7 snap -1 2024-09-13T13:50:55.006 INFO:tasks.rados.rados.0.smithi159.stdout:1141: expect (ObjNum 168 snap 16 seq_num 168) 2024-09-13T13:50:55.006 INFO:tasks.rados.rados.0.smithi159.stdout:1142: write oid 30 current snap is 17 2024-09-13T13:50:55.007 INFO:tasks.rados.rados.0.smithi159.stdout:1142: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-13T13:50:55.008 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:55.014 INFO:tasks.rados.rados.0.smithi159.stdout:1142: writing smithi15938319-30 from 709295 to 1422832 tid 1 2024-09-13T13:50:55.020 INFO:tasks.rados.rados.0.smithi159.stdout:1142: writing smithi15938319-30 from 1859404 to 2609786 tid 2 2024-09-13T13:50:55.021 INFO:tasks.rados.rados.0.smithi159.stdout:1133: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:50:55.029 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:55.029 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:55.047 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:55.045+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:55.051 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:55.051 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:55.071 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:55.072 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:55.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:55.070+0000 7f1003fff640 0 -- 172.21.15.27:0/2663584396 >> v2:172.21.15.159:6805/4073091144 conn(0x7f0fdc033440 msgr2=0x7f0fdc035830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:55.107 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:55.121 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:50:55.121 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 64.891349508999994, 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.122 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:50:55.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:50:55.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.128 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.386611326999997, 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.129 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:50:55.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.386562015000003, 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:50:55.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:50:55.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:50:55.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.386546310999996, 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.143 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:50:55.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:50:55.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:55.149 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.386518635000002, 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:55.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:50:55.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:50:55.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.351301036999999, 2024-09-13T13:50:55.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:55.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.162 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.163 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:55.163 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.344232054999999, 2024-09-13T13:50:55.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:50:55.163 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.163 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:55.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:50:55.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:50:55.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.309693592999999, 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.170 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:55.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:50:55.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:50:55.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:50:55.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.302063750999999, 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:50:55.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:50:55.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.298687379, 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.188 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:50:55.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:50:55.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:50:55.196 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.195828099000003, 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.197 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:50:55.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:50:55.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 37.984804558999997, 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:50:55.203 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:50:55.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 37.973289276000003, 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:50:55.208 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.871980808, 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.212 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:50:55.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:50:55.217 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.218 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.218 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.87191646, 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.219 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:55.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:50:55.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:50:55.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.871888397999999, 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.225 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:50:55.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:50:55.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 13.870206508000001, 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.231 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:50:55.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:50:55.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:50:55.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1810 4.2 4:4d382039:::smithi15938319-28:head [setxattr _header (57) in=7b,truncate 2252138] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 13.863180119999999, 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030247828000000001, 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.239 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1810 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255798+0000", 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255799+0000", 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4399999999999999e-07 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255804+0000", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4909999999999999e-06 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255811+0000", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9970000000000001e-06 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255816+0000", 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:50:55.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255947+0000", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000130171 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255996+0000", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9233000000000001e-05 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260677+0000", 2024-09-13T13:50:55.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046811470000000001 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260726+0000", 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9441000000000003e-05 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.279950+0000", 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019223711000000001 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280000+0000", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9765000000000002e-05 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280101+0000", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010101499999999999 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.282160+0000", 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020587470000000001 2024-09-13T13:50:55.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.285789+0000", 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036294489999999999 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.285900+0000", 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110964 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286046+0000", 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014613300000000001 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1811 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 13.863153488, 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030548815999999999, 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:50:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1811 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255828+0000", 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255835+0000", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2119999999999997e-06 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256027+0000", 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018704600000000001 2024-09-13T13:50:55.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256075+0000", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260766+0000", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046910119999999996 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260793+0000", 2024-09-13T13:50:55.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6594999999999999e-05 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280369+0000", 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019576296 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280418+0000", 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9221999999999997e-05 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286090+0000", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005671873 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286195+0000", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010464299999999999 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286374+0000", 2024-09-13T13:50:55.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179118 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 13.4306783, 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:50:55.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:50:55.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:50:55.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:50:55.258 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:50:55.258 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:50:55.259 INFO:tasks.rados.rados.0.smithi159.stdout:1134: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:50:55.259 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:55.260 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:55.257+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:55.260 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:50:55.288 INFO:tasks.rados.rados.0.smithi159.stdout:1131: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:55.288 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:55.294 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:55.294 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:55.346 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1509507 2024-09-13T13:50:55.360 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:55.359+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:55.365 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:55.365 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:55.365 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:55.365 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:55.374 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1135: finishing write tid 1 to smithi15938319-6 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1135: finishing write tid 2 to smithi15938319-6 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1143: read oid 4 snap -1 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1143: expect deleted 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1131: done (14 left) 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1133: done (13 left) 2024-09-13T13:50:55.433 INFO:tasks.rados.rados.0.smithi159.stdout:1134: done (12 left) 2024-09-13T13:50:55.434 INFO:tasks.rados.rados.0.smithi159.stdout:1144: set_chunk oid 33 target oid 2024-09-13T13:50:55.438 INFO:tasks.rados.rados.0.smithi159.stdout:1144: set_chunk oid 33 offset: 2977280 length: 421888 target oid c1a968810fc052899e6296d9646c25549beb1ff12fca886581fa16e5d896e434 offset: 0 snap 9 2024-09-13T13:50:55.438 INFO:tasks.rados.rados.0.smithi159.stdout:1145: snap_create 2024-09-13T13:50:55.438 INFO:tasks.rados.rados.0.smithi159.stdout:1135: finishing write tid 3 to smithi15938319-6 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1135: finishing write tid 4 to smithi15938319-6 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1135: oid 6 updating version 0 to 28 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1135: oid 6 version 28 is already newer than 27 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1135: oid 6 updating version 28 to 29 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 29 (ObjNum 182 snap 17 seq_num 182) dirty exists 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1135: left oid 6 (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1137: finishing rollback tid 0 to smithi15938319-43 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1136: finishing set_chunk to oid 34 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 35 (ObjNum 106 snap 6 seq_num 106) dirty exists 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1137: finishing rollback tid 1 to smithi15938319-43 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1137: finishing rollback tid 2 to smithi15938319-43 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 112 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1138: finishing write tid 1 to smithi15938319-20 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1138: finishing write tid 2 to smithi15938319-20 2024-09-13T13:50:55.439 INFO:tasks.rados.rados.0.smithi159.stdout:1138: finishing write tid 3 to smithi15938319-20 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: finishing write tid 4 to smithi15938319-20 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: finishing write tid 5 to smithi15938319-20 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: oid 20 updating version 0 to 75 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: oid 20 version 75 is already newer than 73 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: oid 20 version 75 is already newer than 74 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: oid 20 updating version 75 to 76 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 76 (ObjNum 183 snap 17 seq_num 183) dirty exists 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1138: left oid 20 (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:50:55.440 INFO:tasks.rados.rados.0.smithi159.stdout:1139: expect (ObjNum 151 snap 12 seq_num 151) 2024-09-13T13:50:55.448 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1651145 2024-09-13T13:50:55.460 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:55.459+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:55.518 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:55.518 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:55.518 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:55.518 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:55.518 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:55.528 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:50:55.553 INFO:tasks.rados.rados.0.smithi159.stdout:1140: finishing write tid 1 to smithi15938319-36 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: finishing write tid 2 to smithi15938319-36 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: finishing write tid 3 to smithi15938319-36 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: finishing write tid 4 to smithi15938319-36 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: oid 36 updating version 0 to 115 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: oid 36 version 115 is already newer than 114 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: oid 36 updating version 115 to 116 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 116 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1140: left oid 36 (ObjNum 184 snap 17 seq_num 184) 2024-09-13T13:50:55.554 INFO:tasks.rados.rados.0.smithi159.stdout:1141: expect (ObjNum 168 snap 16 seq_num 168) 2024-09-13T13:50:55.561 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 992696 2024-09-13T13:50:55.561 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:55.559+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:55.661 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:55.661 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.926960831999999, 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.662 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:50:55.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:50:55.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.926883431, 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.668 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:50:55.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:50:55.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:50:55.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.926730892999998, 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.675 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:50:55.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:50:55.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.680 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.926624854000003, 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.681 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.686 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.918285775999998, 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:50:55.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:50:55.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.890919066000002, 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.691 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:50:55.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.889152467999999, 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.697 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:50:55.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:50:55.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:50:55.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.884273125999997, 2024-09-13T13:50:55.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:55.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:50:55.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:50:55.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:219 3.2 e41/33 3:650cb820:::smithi15938319-20 low_tier:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:55.709 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.877182142000002, 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02650433, 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781965+0000", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.452e-06 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789891+0000", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079261269999999998 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789958+0000", 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6587999999999993e-05 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789965+0000", 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790010+0000", 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4592000000000003e-05 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790030+0000", 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0630000000000001e-05 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808419+0000", 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018388315999999998 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808468+0000", 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9168000000000001e-05 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.873761401000003, 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:50:55.714 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:55.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:50:55.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.720 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:227 3.2 e41/33 3:626302dd:::smithi15938319-28 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.869249914999997, 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024242393000000001, 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789897+0000", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8999999999999995e-07 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795632+0000", 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057353760000000004 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795680+0000", 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8420000000000001e-05 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795692+0000", 2024-09-13T13:50:55.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1452e-05 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795737+0000", 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5596000000000002e-05 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795751+0000", 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3127e-05 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814104+0000", 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018353103999999999 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814138+0000", 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4527999999999999e-05 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.849504111000002, 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.726 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:50:55.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:50:55.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.846619066000002, 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:50:55.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:50:55.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:50:55.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.844415040999998, 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:55.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:50:55.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.837424396999999, 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.744 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:50:55.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:50:55.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:245 3.2 e41/33 3:60c9a0cb:::smithi15938319-46 low_tier:head v 41'9, mlcod=41'6)", 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.831737410000002, 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024215621999999999, 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827408+0000", 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827409+0000", 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832412+0000", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050024800000000001 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832526+0000", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000113959 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832535+0000", 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0850000000000003e-06 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832620+0000", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.5050999999999996e-05 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.832644+0000", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4012e-05 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851590+0000", 2024-09-13T13:50:55.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018946647 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851624+0000", 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3628999999999999e-05 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:55.753 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.732432783, 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:55.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:50:55.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:50:55.759 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.9088429219999998, 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:50:55.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:50:55.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.9088101320000002, 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:55.766 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:50:55.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:50:55.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:50:55.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:50:55.771 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.7930506250000002, 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:50:55.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:50:55.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.775 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:55.776 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:55.776 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:55.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:50:55.777 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:55.776+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:55.778 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T13:50:55.779 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:50:55.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: oid 30 updating version 0 to 114 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: oid 30 version 114 is already newer than 113 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: oid 30 version 114 is already newer than 112 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 114 (ObjNum 185 snap 17 seq_num 185) dirty exists 2024-09-13T13:50:55.781 INFO:tasks.rados.rados.0.smithi159.stdout:1142: left oid 30 (ObjNum 185 snap 17 seq_num 185) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1135: done (13 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1136: done (12 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1137: done (11 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1138: done (10 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1139: done (9 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1140: done (8 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1141: done (7 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1142: done (6 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1145: done (5 left) 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1146: write (excl) oid 24 current snap is 18 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1146: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1146: writing smithi15938319-24 from 726102 to 1514888 tid 1 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1146: writing smithi15938319-24 from 1917886 to 2528279 tid 2 2024-09-13T13:50:55.782 INFO:tasks.rados.rados.0.smithi159.stdout:1146: writing smithi15938319-24 from 3108206 to 3126102 tid 3 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1143: done (5 left) 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1147: read oid 27 snap -1 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1147: expect deleted 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1148: tier_promote oid 36 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1144: finishing set_chunk to oid 33 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1144: done (6 left) 2024-09-13T13:50:55.783 INFO:tasks.rados.rados.0.smithi159.stdout:1149: set_chunk oid 49 target oid 2024-09-13T13:50:55.783 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:55.777+0000 7f3c37938640 -1 osd.5 78 handle_osd_map injecting map crc failure 2024-09-13T13:50:55.878 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:55.876+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:55.885 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:55.883+0000 7f002dbdb640 -1 osd.15 78 handle_osd_map injecting map crc failure 2024-09-13T13:50:55.924 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:55.924 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:55.924 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:55.924 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:55.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:50:55.979 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:55.977+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:56.036 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.036 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:56.036 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:56.036 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:56.036 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.044 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:50:56.079 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:56.077+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:56.132 INFO:tasks.rados.rados.0.smithi159.stdout:1149: set_chunk oid 49 offset: 798720 length: 1985536 target oid ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0 offset: 0 snap -1 2024-09-13T13:50:56.132 INFO:tasks.rados.rados.0.smithi159.stdout:1150: tier_flush oid 11 2024-09-13T13:50:56.132 INFO:tasks.rados.rados.0.smithi159.stdout:1150: tier_flush oid 11 snap -1 2024-09-13T13:50:56.132 INFO:tasks.rados.rados.0.smithi159.stdout:1146: finishing write tid 1 to smithi15938319-24 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: finishing write tid 2 to smithi15938319-24 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1147: done (7 left) 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1151: write (excl) oid 30 current snap is 18 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: finishing write tid 3 to smithi15938319-24 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: finishing write tid 4 to smithi15938319-24 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: finishing write tid 5 to smithi15938319-24 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: oid 24 updating version 0 to 134 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: oid 24 updating version 134 to 136 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: oid 24 version 136 is already newer than 133 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: oid 24 version 136 is already newer than 135 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 136 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1146: left oid 24 (ObjNum 186 snap 18 seq_num 186) 2024-09-13T13:50:56.133 INFO:tasks.rados.rados.0.smithi159.stdout:1148: got (0) Success 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 116 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: finishing write tid 1 to smithi15938319-2 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: finishing write tid 2 to smithi15938319-2 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: finishing write tid 3 to smithi15938319-2 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: finishing write tid 4 to smithi15938319-2 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: finishing write tid 5 to smithi15938319-2 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: oid 2 updating version 0 to 118 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: oid 2 version 118 is already newer than 116 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: oid 2 version 118 is already newer than 117 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: oid 2 version 118 is already newer than 115 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 118 (ObjNum 181 snap 17 seq_num 181) dirty exists 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1119: left oid 2 (ObjNum 181 snap 17 seq_num 181) 2024-09-13T13:50:56.134 INFO:tasks.rados.rados.0.smithi159.stdout:1121: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:56.157 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.157 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:56.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:56.157 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:56.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.414187503000001, 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:56.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:50:56.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.162 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.408606243999998, 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:50:56.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:50:56.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:50:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.405629632999997, 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:50:56.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.403655462000003, 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.369896361000002, 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:56.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:50:56.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.182 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.364924938999998, 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:50:56.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:50:56.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:56.187 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.361023648, 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:50:56.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:238 3.0 e41/33 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'9)", 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.327611820000001, 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076037040000000002, 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.192 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827891+0000", 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.248e-06 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828642+0000", 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075090499999999995 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828710+0000", 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7478999999999995e-05 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828723+0000", 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3499999999999999e-05 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828822+0000", 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9061999999999994e-05 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828853+0000", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0939999999999999e-05 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835418+0000", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065648920000000001 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835494+0000", 2024-09-13T13:50:56.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5678000000000003e-05 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.320985579999999, 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.197 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:50:56.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.315809737999999, 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:50:56.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.239858660000003, 2024-09-13T13:50:56.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:50:56.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:50:56.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:56.213 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.235057765000001, 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:56.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:50:56.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:50:56.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.219 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:325 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.384396189999997, 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072630530000000002, 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 325 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771107+0000", 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.252e-06 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771114+0000", 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0119999999999999e-06 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771129+0000", 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5131e-05 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771137+0000", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9570000000000002e-06 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771199+0000", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2007000000000004e-05 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771272+0000", 2024-09-13T13:50:56.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3152999999999996e-05 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777134+0000", 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005861387 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778167+0000", 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010329270000000001 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778274+0000", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106827 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778323+0000", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9941999999999997e-05 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778369+0000", 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5457999999999997e-05 2024-09-13T13:50:56.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.080418928, 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.226 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:50:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.957537774999999, 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:50:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:50:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.900557442, 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:50:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:50:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:50:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.895244593000001, 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:50:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:50:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.867466086, 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:50:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:50:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.14.0:186 4.2 e70/63 4:4dc7b2b1:::smithi15938319-14:c v 70'101, mlcod=69'92)", 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 8.8622414490000008, 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0077348039999999996, 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3099999999999998e-07 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293268+0000", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8480000000000003e-06 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293286+0000", 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7802999999999999e-05 2024-09-13T13:50:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293290+0000", 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1710000000000004e-06 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.295355+0000", 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020649639999999999 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.295371+0000", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5777000000000002e-05 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.300956+0000", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055851629999999998 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.300995+0000", 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9647000000000002e-05 2024-09-13T13:50:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.35882583099999998, 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:50:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:50:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:50:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.264 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:50:56.266 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2034057 2024-09-13T13:50:56.267 INFO:tasks.rados.rados.0.smithi159.stdout:1125: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:50:56.267 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:56.265+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:56.343 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1792160 2024-09-13T13:50:56.366 INFO:tasks.rados.rados.0.smithi159.stdout:1150: got (16) Device or resource busy 2024-09-13T13:50:56.366 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 11 v 0 (ObjNum 49 snap 0 seq_num 1953066355) unflushed 16 2024-09-13T13:50:56.366 INFO:tasks.rados.rados.0.smithi159.stdout:1149: finishing set_chunk to oid 49 2024-09-13T13:50:56.366 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 32 (ObjNum 173 snap 16 seq_num 173) dirty exists 2024-09-13T13:50:56.367 INFO:tasks.rados.rados.0.smithi159.stdout:1151: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-13T13:50:56.367 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:56.364+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:56.374 INFO:tasks.rados.rados.0.smithi159.stdout:1151: writing smithi15938319-30 from 742909 to 1206944 tid 1 2024-09-13T13:50:56.374 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.374 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:56.374 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:56.374 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1151: writing smithi15938319-30 from 1976368 to 2730419 tid 2 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1119: done (7 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1121: done (6 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1125: done (5 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1146: done (4 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1148: done (3 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1149: done (2 left) 2024-09-13T13:50:56.383 INFO:tasks.rados.rados.0.smithi159.stdout:1150: done (1 left) 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1152: tier_promote oid 16 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1151: finishing write tid 1 to smithi15938319-30 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1153: read oid 50 snap 13 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1153: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1154: write (excl) oid 18 current snap is 18 2024-09-13T13:50:56.384 INFO:tasks.rados.rados.0.smithi159.stdout:1154: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-13T13:50:56.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:50:56.390 INFO:tasks.rados.rados.0.smithi159.stdout:1154: writing smithi15938319-18 from 759716 to 1299000 tid 1 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1154: writing smithi15938319-18 from 2034850 to 2648912 tid 2 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1155: tier_promote oid 25 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1152: got (0) Success 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 137 (ObjNum 171 snap 16 seq_num 171) dirty exists 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1152: done (4 left) 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1156: delete oid 37 current snap is 18 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1151: finishing write tid 2 to smithi15938319-30 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1151: finishing write tid 3 to smithi15938319-30 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1151: finishing write tid 4 to smithi15938319-30 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1151: oid 30 updating version 0 to 116 2024-09-13T13:50:56.398 INFO:tasks.rados.rados.0.smithi159.stdout:1151: oid 30 updating version 116 to 118 2024-09-13T13:50:56.399 INFO:tasks.rados.rados.0.smithi159.stdout:1151: oid 30 version 118 is already newer than 117 2024-09-13T13:50:56.399 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 118 (ObjNum 187 snap 18 seq_num 187) dirty exists 2024-09-13T13:50:56.399 INFO:tasks.rados.rados.0.smithi159.stdout:1151: left oid 30 (ObjNum 187 snap 18 seq_num 187) 2024-09-13T13:50:56.399 INFO:tasks.rados.rados.0.smithi159.stdout:1153: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:56.399 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 73782 2024-09-13T13:50:56.467 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:56.466+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:56.492 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.492 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:56.492 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:56.492 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:56.492 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.499 INFO:tasks.rados.rados.0.smithi159.stdout:1154: finishing write tid 1 to smithi15938319-18 2024-09-13T13:50:56.499 INFO:tasks.rados.rados.0.smithi159.stdout:1151: done (4 left) 2024-09-13T13:50:56.499 INFO:tasks.rados.rados.0.smithi159.stdout:1153: done (3 left) 2024-09-13T13:50:56.500 INFO:tasks.rados.rados.0.smithi159.stdout:1156: done (2 left) 2024-09-13T13:50:56.500 INFO:tasks.rados.rados.0.smithi159.stdout:1157: write (excl) oid 41 current snap is 18 2024-09-13T13:50:56.500 INFO:tasks.rados.rados.0.smithi159.stdout:1154: finishing write tid 2 to smithi15938319-18 2024-09-13T13:50:56.500 INFO:tasks.rados.rados.0.smithi159.stdout:1154: finishing write tid 3 to smithi15938319-18 2024-09-13T13:50:56.500 INFO:tasks.rados.rados.0.smithi159.stdout:1157: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-13T13:50:56.501 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:50:56.507 INFO:tasks.rados.rados.0.smithi159.stdout:1157: writing smithi15938319-41 from 793330 to 1483112 tid 1 2024-09-13T13:50:56.513 INFO:tasks.rados.rados.0.smithi159.stdout:1157: writing smithi15938319-41 from 2035461 to 2653192 tid 2 2024-09-13T13:50:56.516 INFO:tasks.rados.rados.0.smithi159.stdout:1157: writing smithi15938319-41 from 3101545 to 3193330 tid 3 2024-09-13T13:50:56.517 INFO:tasks.rados.rados.0.smithi159.stdout:1158: set_chunk oid 21 target oid 2024-09-13T13:50:56.528 INFO:tasks.rados.rados.0.smithi159.stdout:1158: set_chunk oid 21 offset: 301056 length: 1037312 target oid 1005830e67a798d39731c15ed3f9e9a0c318c021b125da06e4a74f2b2ff63254 offset: 0 snap -1 2024-09-13T13:50:56.528 INFO:tasks.rados.rados.0.smithi159.stdout:1159: write (excl) oid 2 current snap is 18 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1154: finishing write tid 4 to smithi15938319-18 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1154: oid 18 updating version 0 to 140 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1154: oid 18 version 140 is already newer than 139 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1154: oid 18 version 140 is already newer than 138 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 140 (ObjNum 188 snap 18 seq_num 188) dirty exists 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1154: left oid 18 (ObjNum 188 snap 18 seq_num 188) 2024-09-13T13:50:56.529 INFO:tasks.rados.rados.0.smithi159.stdout:1159: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-13T13:50:56.533 INFO:tasks.rados.rados.0.smithi159.stdout:1159: writing smithi15938319-2 from 410137 to 891521 tid 1 2024-09-13T13:50:56.540 INFO:tasks.rados.rados.0.smithi159.stdout:1159: writing smithi15938319-2 from 1410296 to 2171685 tid 2 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1159: writing smithi15938319-2 from 2728968 to 3210137 tid 3 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1154: done (4 left) 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1160: read oid 19 snap -1 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1160: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1161: write oid 28 current snap is 18 2024-09-13T13:50:56.546 INFO:tasks.rados.rados.0.smithi159.stdout:1161: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-13T13:50:56.551 INFO:tasks.rados.rados.0.smithi159.stdout:1161: writing smithi15938319-28 from 426944 to 983577 tid 1 2024-09-13T13:50:56.557 INFO:tasks.rados.rados.0.smithi159.stdout:1161: writing smithi15938319-28 from 1468778 to 2090178 tid 2 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1161: writing smithi15938319-28 from 2872744 to 3226944 tid 3 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1162: read oid 48 snap -1 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1162: expect (ObjNum 162 snap 13 seq_num 162) 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1163: write (excl) oid 42 current snap is 18 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1155: got (0) Success 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 61 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1157: finishing write tid 1 to smithi15938319-41 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1157: finishing write tid 2 to smithi15938319-41 2024-09-13T13:50:56.565 INFO:tasks.rados.rados.0.smithi159.stdout:1157: finishing write tid 3 to smithi15938319-41 2024-09-13T13:50:56.566 INFO:tasks.rados.rados.0.smithi159.stdout:1163: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-13T13:50:56.568 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:56.567+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:56.570 INFO:tasks.rados.rados.0.smithi159.stdout:1163: writing smithi15938319-42 from 443751 to 1075633 tid 1 2024-09-13T13:50:56.578 INFO:tasks.rados.rados.0.smithi159.stdout:1163: writing smithi15938319-42 from 1810907 to 2575965 tid 2 2024-09-13T13:50:56.583 INFO:tasks.rados.rados.0.smithi159.stdout:1163: writing smithi15938319-42 from 3067461 to 3243751 tid 3 2024-09-13T13:50:56.583 INFO:tasks.rados.rados.0.smithi159.stdout:1155: done (7 left) 2024-09-13T13:50:56.583 INFO:tasks.rados.rados.0.smithi159.stdout:1164: tier_promote oid 1 2024-09-13T13:50:56.583 INFO:tasks.rados.rados.0.smithi159.stdout:1157: finishing write tid 4 to smithi15938319-41 2024-09-13T13:50:56.583 INFO:tasks.rados.rados.0.smithi159.stdout:1165: read oid 20 snap -1 2024-09-13T13:50:56.584 INFO:tasks.rados.rados.0.smithi159.stdout:1165: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:50:56.584 INFO:tasks.rados.rados.0.smithi159.stdout:1166: read oid 45 snap 13 2024-09-13T13:50:56.584 INFO:tasks.rados.rados.0.smithi159.stdout:1166: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:56.584 INFO:tasks.rados.rados.0.smithi159.stdout:1167: tier_promote oid 36 2024-09-13T13:50:56.584 INFO:tasks.rados.rados.0.smithi159.stdout:1168: set_chunk oid 16 target oid 2024-09-13T13:50:56.593 INFO:tasks.rados.rados.0.smithi159.stdout:1168: set_chunk oid 16 offset: 69120 length: 499200 target oid 3b0b4df9ca95d89f245ef6d53fd41c5e84cb5ff15e41ba87c70577af538168ed offset: 0 snap -1 2024-09-13T13:50:56.593 INFO:tasks.rados.rados.0.smithi159.stdout:1169: read oid 39 snap -1 2024-09-13T13:50:56.593 INFO:tasks.rados.rados.0.smithi159.stdout:1169: expect deleted 2024-09-13T13:50:56.593 INFO:tasks.rados.rados.0.smithi159.stdout:1170: snap_remove snap 9 2024-09-13T13:50:56.619 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:56.619 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.354913789000001, 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.620 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.621 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:50:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.623 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.624 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:50:56.625 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:50:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:50:56.626 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:50:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.627 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:50:56.628 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:50:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.629 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.353960290000003, 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.630 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.631 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.632 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.633 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:50:56.634 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:50:56.635 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:50:56.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.639 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.127710315999998, 2024-09-13T13:50:56.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:50:56.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:50:56.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:50:56.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:50:56.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:127 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'6, mlcod=41'3)", 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.126791312999998, 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029472949999999999, 2024-09-13T13:50:56.647 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489858+0000", 2024-09-13T13:50:56.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489859+0000", 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8599999999999998e-07 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489864+0000", 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1209999999999998e-06 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489892+0000", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8059e-05 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489899+0000", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4429999999999997e-06 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489946+0000", 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7018000000000003e-05 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489965+0000", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8419000000000001e-05 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492765+0000", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028008830000000001 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492805+0000", 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9666000000000003e-05 2024-09-13T13:50:56.651 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:262 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'7, mlcod=41'5)", 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.704994935999999, 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024065520000000002, 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.652 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911654+0000", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911655+0000", 2024-09-13T13:50:56.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1790000000000001e-06 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911665+0000", 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7059999999999999e-06 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911706+0000", 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0985000000000001e-05 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911715+0000", 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4319999999999999e-06 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911772+0000", 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7462e-05 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.911801+0000", 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9023e-05 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.914031+0000", 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022304650000000001 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.914061+0000", 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9300000000000001e-05 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.656 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.843124029999998, 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:50:56.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1026 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'15, mlcod=42'12)", 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.836437598000003, 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031666730000000001, 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.661 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:56.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780212+0000", 2024-09-13T13:50:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200000000000002e-07 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780461+0000", 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00024905400000000002 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780488+0000", 2024-09-13T13:50:56.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6423000000000001e-05 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780497+0000", 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.918e-06 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780550+0000", 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3195000000000001e-05 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.780574+0000", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3734e-05 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783341+0000", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027677679999999999 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783378+0000", 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6769e-05 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.667 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1027 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'16, mlcod=42'12)", 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.835620872, 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024073179999999999, 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781028+0000", 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781029+0000", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999998e-07 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781037+0000", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9109999999999993e-06 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781075+0000", 2024-09-13T13:50:56.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8087000000000003e-05 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781082+0000", 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5030000000000002e-06 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781131+0000", 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9073999999999998e-05 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.781177+0000", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6338999999999998e-05 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783410+0000", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022328259999999998 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.783436+0000", 2024-09-13T13:50:56.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5758e-05 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 28.397758249999999, 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.673 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:50:56.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.677 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1219 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'20, mlcod=52'17)", 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 28.39690246, 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031764929999999999, 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.678 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219747+0000", 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219751+0000", 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9310000000000001e-06 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219767+0000", 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6053999999999999e-05 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219771+0000", 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4130000000000002e-06 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219981+0000", 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00021069900000000001 2024-09-13T13:50:56.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219994+0000", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3174e-05 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222896+0000", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0029013910000000001 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222923+0000", 2024-09-13T13:50:56.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7382e-05 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:91 4.8 e52/34 4:165e03eb:::smithi15938319-32:head v 52'22, mlcod=48'19)", 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 28.379912750999999, 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030596489999999998, 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.683 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236736+0000", 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.236737+0000", 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.79e-07 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237317+0000", 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057988500000000004 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237354+0000", 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7883000000000003e-05 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237362+0000", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4660000000000002e-06 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237421+0000", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8588999999999998e-05 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.237446+0000", 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5918000000000002e-05 2024-09-13T13:50:56.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239751+0000", 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023044540000000001 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.239796+0000", 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5175e-05 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1296 4.8 e53/34 4:165e03eb:::smithi15938319-32:head v 53'25, mlcod=52'22)", 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:29.250022+0000", 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.366626847999999, 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032048329999999998, 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250022+0000", 2024-09-13T13:50:56.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250022+0000", 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250024+0000", 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4109999999999999e-06 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250367+0000", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000343741 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250400+0000", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2843999999999999e-05 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250409+0000", 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4160000000000004e-06 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250470+0000", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1628e-05 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.250492+0000", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1489999999999999e-05 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.253182+0000", 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026898550000000001 2024-09-13T13:50:56.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:29.253227+0000", 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5448000000000003e-05 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2049 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'52, mlcod=66'48)", 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.2931068630000002, 2024-09-13T13:50:56.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003216062, 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323795+0000", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025235600000000002 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323833+0000", 2024-09-13T13:50:56.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8479000000000002e-05 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323843+0000", 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0142999999999999e-05 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323913+0000", 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9250000000000003e-05 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323940+0000", 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7047999999999999e-05 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326715+0000", 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027753679999999998 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326758+0000", 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3248999999999999e-05 2024-09-13T13:50:56.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:50:56.697 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:50:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.2902134570000001, 2024-09-13T13:50:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:50:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:50:56.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:50:56.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.2894645679999996, 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.704 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:50:56.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:50:56.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2123 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'60, mlcod=70'56)", 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.8769295860000001, 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035624979999999999, 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:50:56.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3299999999999998e-07 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740298+0000", 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057840599999999995 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740767+0000", 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046913499999999997 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740784+0000", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6487999999999999e-05 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740877+0000", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2666999999999999e-05 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740903+0000", 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6177999999999999e-05 2024-09-13T13:50:56.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743246+0000", 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023431390000000002 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743282+0000", 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6152000000000002e-05 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.8711303859999999, 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:50:56.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.8660020280000005, 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:50:56.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:50:56.723 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.8650904720000003, 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:50:56.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:50:56.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:134 4.8 e78/34 4:1b24d731:::smithi15938319-13:11 v 78'71, mlcod=75'65)", 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.99788071099999998, 2024-09-13T13:50:56.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033973689999999999, 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618769+0000", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618785+0000", 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6127999999999999e-05 2024-09-13T13:50:56.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618814+0000", 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9110000000000001e-05 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618821+0000", 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3130000000000004e-06 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:56.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618874+0000", 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3596e-05 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618890+0000", 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6203999999999999e-05 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622135+0000", 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032444000000000001 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:56.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622166+0000", 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0969000000000001e-05 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:56.733 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.735 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:50:56.735 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:56.733+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:56.761 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:50:56.761 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T13:50:56.762 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:56.760+0000 7f1f37764640 -1 osd.4 79 handle_osd_map injecting map crc failure 2024-09-13T13:50:56.769 INFO:tasks.rados.rados.0.smithi159.stdout:1170: done (13 left) 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1171: read oid 34 snap -1 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1171: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1172: read oid 32 snap -1 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1172: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1173: read oid 38 snap 6 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1173: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1157: finishing write tid 5 to smithi15938319-41 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1157: oid 41 updating version 0 to 120 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1157: oid 41 version 120 is already newer than 119 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1157: oid 41 version 120 is already newer than 118 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:1157: oid 41 updating version 120 to 121 2024-09-13T13:50:56.770 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 121 (ObjNum 190 snap 18 seq_num 190) dirty exists 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1157: left oid 41 (ObjNum 190 snap 18 seq_num 190) 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1158: finishing set_chunk to oid 21 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 301056 length: 1037312 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 151 snap 12 seq_num 151) dirty exists 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: finishing write tid 1 to smithi15938319-2 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: finishing write tid 2 to smithi15938319-2 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: finishing write tid 3 to smithi15938319-2 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: finishing write tid 4 to smithi15938319-2 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: finishing write tid 5 to smithi15938319-2 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: oid 2 updating version 0 to 122 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: oid 2 updating version 122 to 124 2024-09-13T13:50:56.771 INFO:tasks.rados.rados.0.smithi159.stdout:1159: oid 2 updating version 124 to 125 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1159: oid 2 version 125 is already newer than 123 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 125 (ObjNum 191 snap 18 seq_num 191) dirty exists 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1159: left oid 2 (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1157: done (15 left) 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1158: done (14 left) 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1159: done (13 left) 2024-09-13T13:50:56.772 INFO:tasks.rados.rados.0.smithi159.stdout:1174: set_chunk oid 15 target oid 2024-09-13T13:50:56.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T13:50:56.792 INFO:tasks.rados.rados.0.smithi159.stdout:1174: set_chunk oid 15 offset: 822784 length: 1120256 target oid 3634ac6462bf9cb96fe06c81293bb3c6c0e95892f99c8f55c0a14944e78c533e offset: 0 snap -1 2024-09-13T13:50:56.792 INFO:tasks.rados.rados.0.smithi159.stdout:1175: write (excl) oid 5 current snap is 18 2024-09-13T13:50:56.792 INFO:tasks.rados.rados.0.smithi159.stdout:1160: expect (ObjNum 89 snap 5 seq_num 89) 2024-09-13T13:50:56.792 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:56.835 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:56.834+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:56.877 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:56.877 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:56.877 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:50:56.877 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:56.886 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:50:56.937 INFO:tasks.rados.rados.0.smithi159.stdout:1161: finishing write tid 1 to smithi15938319-28 2024-09-13T13:50:56.937 INFO:tasks.rados.rados.0.smithi159.stdout:1161: finishing write tid 2 to smithi15938319-28 2024-09-13T13:50:56.937 INFO:tasks.rados.rados.0.smithi159.stdout:1161: finishing write tid 3 to smithi15938319-28 2024-09-13T13:50:56.937 INFO:tasks.rados.rados.0.smithi159.stdout:1175: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-13T13:50:56.938 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:56.935+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:56.943 INFO:tasks.rados.rados.0.smithi159.stdout:1175: writing smithi15938319-5 from 460558 to 1167689 tid 1 2024-09-13T13:50:56.950 INFO:tasks.rados.rados.0.smithi159.stdout:1175: writing smithi15938319-5 from 1869389 to 2378105 tid 2 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1175: writing smithi15938319-5 from 3094884 to 3260558 tid 3 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1160: done (14 left) 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: finishing write tid 4 to smithi15938319-28 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: finishing write tid 5 to smithi15938319-28 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: oid 28 updating version 0 to 128 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: oid 28 updating version 128 to 129 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: oid 28 version 129 is already newer than 127 2024-09-13T13:50:56.955 INFO:tasks.rados.rados.0.smithi159.stdout:1161: oid 28 updating version 129 to 130 2024-09-13T13:50:56.956 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 130 (ObjNum 192 snap 18 seq_num 192) dirty exists 2024-09-13T13:50:56.956 INFO:tasks.rados.rados.0.smithi159.stdout:1161: left oid 28 (ObjNum 192 snap 18 seq_num 192) 2024-09-13T13:50:56.956 INFO:tasks.rados.rados.0.smithi159.stdout:1162: expect (ObjNum 162 snap 13 seq_num 162) 2024-09-13T13:50:56.999 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 628888 2024-09-13T13:50:57.031 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:57.031 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:50:57.031 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:50:57.031 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:50:57.031 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:57.041 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:50:57.042 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:57.036+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:57.080 INFO:tasks.rados.rados.0.smithi159.stdout:1163: finishing write tid 1 to smithi15938319-42 2024-09-13T13:50:57.080 INFO:tasks.rados.rados.0.smithi159.stdout:1164: got (0) Success 2024-09-13T13:50:57.080 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 86 (ObjNum 176 snap 16 seq_num 176) dirty exists 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1163: finishing write tid 2 to smithi15938319-42 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1167: got (0) Success 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 116 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1163: finishing write tid 3 to smithi15938319-42 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1163: finishing write tid 4 to smithi15938319-42 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1176: read oid 27 snap -1 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1176: expect deleted 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1161: done (14 left) 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1162: done (13 left) 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1164: done (12 left) 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1167: done (11 left) 2024-09-13T13:50:57.081 INFO:tasks.rados.rados.0.smithi159.stdout:1177: write oid 43 current snap is 18 2024-09-13T13:50:57.082 INFO:tasks.rados.rados.0.smithi159.stdout:1177: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-13T13:50:57.086 INFO:tasks.rados.rados.0.smithi159.stdout:1177: writing smithi15938319-43 from 477365 to 1259745 tid 1 2024-09-13T13:50:57.094 INFO:tasks.rados.rados.0.smithi159.stdout:1177: writing smithi15938319-43 from 1927871 to 2696598 tid 2 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1177: writing smithi15938319-43 from 3122307 to 3277365 tid 3 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1178: set_chunk oid 21 target oid 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: finishing write tid 5 to smithi15938319-42 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: oid 42 updating version 0 to 77 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: oid 42 version 77 is already newer than 75 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: oid 42 version 77 is already newer than 76 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: oid 42 version 77 is already newer than 74 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 77 (ObjNum 193 snap 18 seq_num 193) dirty exists 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1163: left oid 42 (ObjNum 193 snap 18 seq_num 193) 2024-09-13T13:50:57.098 INFO:tasks.rados.rados.0.smithi159.stdout:1165: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:50:57.100 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 568860 2024-09-13T13:50:57.139 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:57.138+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:57.186 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 68.642203777999995, 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:50:57.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:50:57.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.982761711999999, 2024-09-13T13:50:57.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:50:57.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:50:57.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.981373488000003, 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.196 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:50:57.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:50:57.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.980671485000002, 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:50:57.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:96 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'63)", 2024-09-13T13:50:57.205 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.980185681999998, 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050574289999999996, 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204033+0000", 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204133+0000", 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9450000000000005e-05 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204163+0000", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0018999999999998e-05 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204168+0000", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5539999999999997e-06 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206261+0000", 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020930889999999998 2024-09-13T13:50:57.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206274+0000", 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3018e-05 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209065+0000", 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027905909999999998 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209090+0000", 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4896999999999999e-05 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.587069147999998, 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:50:57.210 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:50:57.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:50:57.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:116 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'86, mlcod=25'80)", 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.586524709000003, 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051291219999999998, 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.215 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597694+0000", 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597697+0000", 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6440000000000002e-06 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597713+0000", 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5699999999999999e-05 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597718+0000", 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1669999999999998e-06 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599712+0000", 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019937420000000002 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599730+0000", 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8091e-05 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602796+0000", 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003066306 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602823+0000", 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6112999999999999e-05 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.220 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.580945345000003, 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:50:57.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:50:57.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:21 4.d e41/34 4:b771c1ad:::smithi15938319-6:head v 41'1, mlcod=0'0)", 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.003745459000001, 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049861549999999999, 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180473+0000", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.180474+0000", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5799999999999998e-07 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181463+0000", 2024-09-13T13:50:57.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009894490000000001 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181490+0000", 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7053999999999999e-05 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181498+0000", 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8460000000000008e-06 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181554+0000", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6335000000000002e-05 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.181571+0000", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6328e-05 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.185401+0000", 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038303349999999998 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.185459+0000", 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7849999999999997e-05 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.230 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.267388189000002, 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.231 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:57.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:50:57.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:50:57.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.413549441000001, 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.238 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:50:57.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:50:57.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.244 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:812 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 2412870~427480,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 38.994199174000002, 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054419789999999996, 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 812 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190022+0000", 2024-09-13T13:50:57.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190034+0000", 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190058+0000", 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190071+0000", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3861e-05 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190156+0000", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4179999999999997e-05 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190164+0000", 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-09-13T13:50:57.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194705+0000", 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045405109999999997 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194761+0000", 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6140999999999997e-05 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195461+0000", 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070009399999999996 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 38.994114572000001, 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.250 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:57.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:50:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 38.994066621999998, 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:50:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:50:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.305603695000002, 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:50:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 11.609996568, 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:50:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:50:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:50:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:50:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.8822942939999994, 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:50:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:50:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:50:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:50:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.856784889, 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:50:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:50:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.8561191519999998, 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:50:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:50:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.8560781360000007, 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:50:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:50:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:50:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:50:57.292 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:50:57.293 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:50:57.292+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:50:57.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:50:57.294 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.294 INFO:tasks.rados.rados.0.smithi159.stdout:1166: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:50:57.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:50:57.364 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3091962 2024-09-13T13:50:57.374 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1594870 2024-09-13T13:50:57.394 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:57.394+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:50:57.403 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:50:57.403 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:50:57.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:50:57.403 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:50:57.406 INFO:tasks.rados.rados.0.smithi159.stdout:1168: finishing set_chunk to oid 16 2024-09-13T13:50:57.406 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 137 (ObjNum 171 snap 16 seq_num 171) dirty exists 2024-09-13T13:50:57.406 INFO:tasks.rados.rados.0.smithi159.stdout:1172: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:50:57.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:50:57.460 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.490 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:50:57.495 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:57.493+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:57.513 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:50:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:50:57.519 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.532 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:57.532 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:57.535 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:50:57.535 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:50:57.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:50:57.556 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:50:57.557 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:50:57.577 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:50:57.578 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:50:57.579 INFO:tasks.rados.rados.0.smithi159.stdout:1171: expect (ObjNum 106 snap 6 seq_num 106) 2024-09-13T13:50:57.579 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.596 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:57.594+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:57.599 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:50:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:50:57.621 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:50:57.621 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:50:57.642 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:50:57.642 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:50:57.649 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:57.649 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:57.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:50:57.663 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:50:57.663 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:50:57.685 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:50:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:50:57.697 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:57.695+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:50:57.706 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:50:57.707 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:50:57.727 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:50:57.728 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:50:57.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:50:57.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:50:57.751 INFO:tasks.rados.rados.0.smithi159.stdout:1173: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-13T13:50:57.752 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:50:57.753+0000 7f32b1de3640 0 -- 172.21.15.27:0/3610564200 >> v2:172.21.15.100:6800/450451503 conn(0x7f32ac21ab60 msgr2=0x7f32ac21cf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:50:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T13:50:57.772 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:57.772 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:57.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:50:57.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:50:57.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:50:57.784 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 685646 2024-09-13T13:50:57.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T13:50:57.797 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:57.797+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:57.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:50:57.802 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:50:57.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:50:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:50:57.833 INFO:tasks.rados.rados.0.smithi159.stdout:1174: finishing set_chunk to oid 15 2024-09-13T13:50:57.834 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 92 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1178: set_chunk oid 21 offset: 585728 length: 1740800 target oid b44430f69bc78435381ac552ba016087abc7e93c2e60092acebbc1f1136e4822 offset: 0 snap 16 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1163: done (12 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1165: done (11 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1166: done (10 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1168: done (9 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1169: done (8 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1171: done (7 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1172: done (6 left) 2024-09-13T13:50:57.850 INFO:tasks.rados.rados.0.smithi159.stdout:1173: done (5 left) 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1174: done (4 left) 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1179: read oid 50 snap 13 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1179: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1180: write oid 42 current snap is 18 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: finishing write tid 1 to smithi15938319-5 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: finishing write tid 2 to smithi15938319-5 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: finishing write tid 3 to smithi15938319-5 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: finishing write tid 4 to smithi15938319-5 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: finishing write tid 5 to smithi15938319-5 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: oid 5 updating version 0 to 84 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: oid 5 updating version 84 to 86 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: oid 5 version 86 is already newer than 85 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:1175: oid 5 version 86 is already newer than 83 2024-09-13T13:50:57.851 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 86 (ObjNum 194 snap 18 seq_num 194) dirty exists 2024-09-13T13:50:57.852 INFO:tasks.rados.rados.0.smithi159.stdout:1175: left oid 5 (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:50:57.852 INFO:tasks.rados.rados.0.smithi159.stdout:1180: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-13T13:50:57.854 INFO:tasks.rados.rados.0.smithi159.stdout:1180: writing smithi15938319-42 from 494172 to 951801 tid 1 2024-09-13T13:50:57.860 INFO:tasks.rados.rados.0.smithi159.stdout:1180: writing smithi15938319-42 from 1586353 to 2098738 tid 2 2024-09-13T13:50:57.867 INFO:tasks.rados.rados.0.smithi159.stdout:1180: writing smithi15938319-42 from 2749730 to 3294172 tid 3 2024-09-13T13:50:57.867 INFO:tasks.rados.rados.0.smithi159.stdout:1175: done (5 left) 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1181: read oid 18 snap 17 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1181: expect (ObjNum 123 snap 8 seq_num 123) 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1182: rollback oid 11 current snap is 18 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 11 to 6 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: finishing write tid 1 to smithi15938319-43 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: finishing write tid 2 to smithi15938319-43 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: finishing write tid 3 to smithi15938319-43 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: finishing write tid 4 to smithi15938319-43 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: finishing write tid 5 to smithi15938319-43 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: oid 43 updating version 0 to 130 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: oid 43 version 130 is already newer than 127 2024-09-13T13:50:57.868 INFO:tasks.rados.rados.0.smithi159.stdout:1177: oid 43 version 130 is already newer than 129 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:1177: oid 43 version 130 is already newer than 128 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 130 (ObjNum 195 snap 18 seq_num 195) dirty exists 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:1177: left oid 43 (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:1178: finishing set_chunk to oid 21 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 585728 length: 1740800 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 0 (ObjNum 151 snap 12 seq_num 151) dirty exists 2024-09-13T13:50:57.869 INFO:tasks.rados.rados.0.smithi159.stdout:1179: expect (ObjNum 143 snap 11 seq_num 143) 2024-09-13T13:50:57.899 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:57.898+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:57.912 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:57.912 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:57.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:50:57.939 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1180: finishing write tid 1 to smithi15938319-42 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1180: finishing write tid 2 to smithi15938319-42 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1182: finishing rollback tid 1 to smithi15938319-11 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1182: finishing rollback tid 2 to smithi15938319-11 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 131 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1180: finishing write tid 3 to smithi15938319-42 2024-09-13T13:50:57.975 INFO:tasks.rados.rados.0.smithi159.stdout:1180: finishing write tid 4 to smithi15938319-42 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: finishing write tid 5 to smithi15938319-42 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: oid 42 updating version 0 to 85 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: oid 42 version 85 is already newer than 83 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: oid 42 version 85 is already newer than 82 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: oid 42 version 85 is already newer than 84 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 85 (ObjNum 196 snap 18 seq_num 196) dirty exists 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1180: left oid 42 (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:1181: expect (ObjNum 123 snap 8 seq_num 123) 2024-09-13T13:50:57.976 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:58.000 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:57.998+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:58.030 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.030 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:50:58.070 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:50:58.069+0000 7f586a715640 -1 osd.9 80 handle_osd_map injecting map crc failure 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1176: done (6 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1177: done (5 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1178: done (4 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1179: done (3 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1180: done (2 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1181: done (1 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1182: done (0 left) 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1183: write (excl) oid 36 current snap is 18 2024-09-13T13:50:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1183: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-13T13:50:58.095 INFO:tasks.rados.rados.0.smithi159.stdout:1183: writing smithi15938319-36 from 510979 to 1043857 tid 1 2024-09-13T13:50:58.100 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:50:58.098+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:50:58.102 INFO:tasks.rados.rados.0.smithi159.stdout:1183: writing smithi15938319-36 from 1528482 to 2300878 tid 2 2024-09-13T13:50:58.107 INFO:tasks.rados.rados.0.smithi159.stdout:1183: writing smithi15938319-36 from 3060800 to 3310979 tid 3 2024-09-13T13:50:58.107 INFO:tasks.rados.rados.0.smithi159.stdout:1184: rollback oid 33 current snap is 18 2024-09-13T13:50:58.108 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 33 to 6 2024-09-13T13:50:58.108 INFO:tasks.rados.rados.0.smithi159.stdout:1183: finishing write tid 1 to smithi15938319-36 2024-09-13T13:50:58.108 INFO:tasks.rados.rados.0.smithi159.stdout:1183: finishing write tid 2 to smithi15938319-36 2024-09-13T13:50:58.108 INFO:tasks.rados.rados.0.smithi159.stdout:1185: tier_promote oid 29 2024-09-13T13:50:58.108 INFO:tasks.rados.rados.0.smithi159.stdout:1186: set_chunk oid 30 target oid 2024-09-13T13:50:58.129 INFO:tasks.rados.rados.0.smithi159.stdout:1186: set_chunk oid 30 offset: 400896 length: 2041856 target oid 8b7df16124d62a6d2a856cb89cc0860ee94b5bfa50169f93fc2e1cd735a3b3c6 offset: 0 snap -1 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1187: tier_evict oid 37 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1187: tier_evict oid 37 snap -1 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1188: tier_flush oid 32 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1188: tier_flush oid 32 snap -1 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: finishing write tid 3 to smithi15938319-36 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: finishing write tid 4 to smithi15938319-36 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: finishing write tid 5 to smithi15938319-36 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: oid 36 updating version 0 to 134 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: oid 36 updating version 134 to 135 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: oid 36 version 135 is already newer than 132 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: oid 36 version 135 is already newer than 133 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 135 (ObjNum 197 snap 18 seq_num 197) dirty exists 2024-09-13T13:50:58.130 INFO:tasks.rados.rados.0.smithi159.stdout:1183: left oid 36 (ObjNum 197 snap 18 seq_num 197) 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1184: finishing rollback tid 0 to smithi15938319-33 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1183: done (5 left) 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1189: write oid 35 current snap is 18 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1185: got (0) Success 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 57 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1184: finishing rollback tid 1 to smithi15938319-33 2024-09-13T13:50:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1189: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-13T13:50:58.135 INFO:tasks.rados.rados.0.smithi159.stdout:1189: writing smithi15938319-35 from 527786 to 1252266 tid 1 2024-09-13T13:50:58.140 INFO:tasks.rados.rados.0.smithi159.stdout:1189: writing smithi15938319-35 from 1703317 to 2219371 tid 2 2024-09-13T13:50:58.146 INFO:tasks.rados.rados.0.smithi159.stdout:1189: writing smithi15938319-35 from 2804576 to 3239611 tid 3 2024-09-13T13:50:58.146 INFO:tasks.rados.rados.0.smithi159.stdout:1185: done (5 left) 2024-09-13T13:50:58.146 INFO:tasks.rados.rados.0.smithi159.stdout:1190: set_chunk oid 29 target oid 2024-09-13T13:50:58.162 INFO:tasks.rados.rados.0.smithi159.stdout:1190: set_chunk oid 29 offset: 1259008 length: 1644544 target oid 78c236af8b2637b2f6ef7a557b64cad116398318ed587bda1f79c959554d4e26 offset: 0 snap -1 2024-09-13T13:50:58.162 INFO:tasks.rados.rados.0.smithi159.stdout:1191: tier_flush oid 36 2024-09-13T13:50:58.162 INFO:tasks.rados.rados.0.smithi159.stdout:1191: tier_flush oid 36 snap -1 2024-09-13T13:50:58.162 INFO:tasks.rados.rados.0.smithi159.stdout:1184: finishing rollback tid 2 to smithi15938319-33 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 139 (ObjNum 101 snap 6 seq_num 101) dirty exists 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1184: done (6 left) 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1192: rollback oid 46 current snap is 18 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 46 to 17 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1187: got (22) Invalid argument 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1188: got (16) Device or resource busy 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 32 v 56 (ObjNum 161 snap 13 seq_num 161) unflushed 13 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1186: finishing set_chunk to oid 30 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 118 (ObjNum 187 snap 18 seq_num 187) dirty exists 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1189: finishing write tid 1 to smithi15938319-35 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1189: finishing write tid 2 to smithi15938319-35 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1186: done (6 left) 2024-09-13T13:50:58.163 INFO:tasks.rados.rados.0.smithi159.stdout:1187: done (5 left) 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1188: done (4 left) 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1193: read oid 49 snap 16 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1193: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1194: write oid 16 current snap is 18 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: finishing write tid 3 to smithi15938319-35 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: finishing write tid 4 to smithi15938319-35 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: finishing write tid 5 to smithi15938319-35 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: oid 35 updating version 0 to 164 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: oid 35 version 164 is already newer than 161 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: oid 35 version 164 is already newer than 163 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: oid 35 version 164 is already newer than 162 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 164 (ObjNum 198 snap 18 seq_num 198) dirty exists 2024-09-13T13:50:58.164 INFO:tasks.rados.rados.0.smithi159.stdout:1189: left oid 35 (ObjNum 198 snap 18 seq_num 198) 2024-09-13T13:50:58.165 INFO:tasks.rados.rados.0.smithi159.stdout:1194: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-13T13:50:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:1194: writing smithi15938319-16 from 544593 to 1344322 tid 1 2024-09-13T13:50:58.175 INFO:tasks.rados.rados.0.smithi159.stdout:1194: writing smithi15938319-16 from 1761799 to 2537864 tid 2 2024-09-13T13:50:58.179 INFO:tasks.rados.rados.0.smithi159.stdout:1194: writing smithi15938319-16 from 2948352 to 3344593 tid 3 2024-09-13T13:50:58.179 INFO:tasks.rados.rados.0.smithi159.stdout:1189: done (5 left) 2024-09-13T13:50:58.179 INFO:tasks.rados.rados.0.smithi159.stdout:1195: set_chunk oid 11 target oid 2024-09-13T13:50:58.189 INFO:tasks.rados.rados.0.smithi159.stdout:1195: set_chunk oid 11 offset: 664064 length: 1369600 target oid 9c834470b24a6778e120f7314d88f2fd06904f1810763663e4b00ad4d6242f8a offset: 0 snap -1 2024-09-13T13:50:58.189 INFO:tasks.rados.rados.0.smithi159.stdout:1196: read oid 13 snap -1 2024-09-13T13:50:58.189 INFO:tasks.rados.rados.0.smithi159.stdout:1196: expect (ObjNum 167 snap 16 seq_num 167) 2024-09-13T13:50:58.189 INFO:tasks.rados.rados.0.smithi159.stdout:1197: set_chunk oid 14 target oid 2024-09-13T13:50:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.201 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:58.199+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:58.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:50:58.212 INFO:tasks.rados.rados.0.smithi159.stdout:1197: set_chunk oid 14 offset: 510464 length: 2351104 target oid b223cd215192513b337306d70fc721a96aaacce6a51ab3616964a663c1e0901c offset: 0 snap -1 2024-09-13T13:50:58.212 INFO:tasks.rados.rados.0.smithi159.stdout:1198: read oid 37 snap -1 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:1198: expect deleted 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:1199: write (excl) oid 38 current snap is 18 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:1190: finishing set_chunk to oid 29 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 1259008 length: 1644544 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 0 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:50:58.213 INFO:tasks.rados.rados.0.smithi159.stdout:1199: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-13T13:50:58.217 INFO:tasks.rados.rados.0.smithi159.stdout:1199: writing smithi15938319-38 from 561400 to 1036378 tid 1 2024-09-13T13:50:58.224 INFO:tasks.rados.rados.0.smithi159.stdout:1199: writing smithi15938319-38 from 1820281 to 2340004 tid 2 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1199: writing smithi15938319-38 from 2859422 to 3317001 tid 3 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1190: done (9 left) 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1200: tier_promote oid 24 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1201: read oid 28 snap 13 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1201: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:1191: got (16) Device or resource busy 2024-09-13T13:50:58.230 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 36 v 135 (ObjNum 197 snap 18 seq_num 197) unflushed 18 2024-09-13T13:50:58.231 INFO:tasks.rados.rados.0.smithi159.stdout:1192: finishing rollback tid 1 to smithi15938319-46 2024-09-13T13:50:58.231 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 63 (ObjNum 129 snap 0 seq_num 1540206104) dirty dne 2024-09-13T13:50:58.231 INFO:tasks.rados.rados.0.smithi159.stdout:1193: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:50:58.274 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:50:58.273+0000 7f8f58c61640 -1 osd.6 80 handle_osd_map injecting map crc failure 2024-09-13T13:50:58.280 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 755371 2024-09-13T13:50:58.302 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:58.300+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:58.314 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.314 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.322 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:50:58.357 INFO:tasks.rados.rados.0.smithi159.stdout:1194: finishing write tid 1 to smithi15938319-16 2024-09-13T13:50:58.357 INFO:tasks.rados.rados.0.smithi159.stdout:1194: finishing write tid 2 to smithi15938319-16 2024-09-13T13:50:58.357 INFO:tasks.rados.rados.0.smithi159.stdout:1194: finishing write tid 3 to smithi15938319-16 2024-09-13T13:50:58.357 INFO:tasks.rados.rados.0.smithi159.stdout:1194: finishing write tid 4 to smithi15938319-16 2024-09-13T13:50:58.357 INFO:tasks.rados.rados.0.smithi159.stdout:1194: finishing write tid 5 to smithi15938319-16 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: oid 16 updating version 0 to 168 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: oid 16 updating version 168 to 169 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: oid 16 version 169 is already newer than 167 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: oid 16 version 169 is already newer than 166 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 169 (ObjNum 199 snap 18 seq_num 199) dirty exists 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: left oid 16 (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1202: read oid 50 snap -1 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1202: expect deleted 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1191: done (11 left) 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1192: done (10 left) 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1193: done (9 left) 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1194: done (8 left) 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1203: snap_create 2024-09-13T13:50:58.358 INFO:tasks.rados.rados.0.smithi159.stdout:1196: expect (ObjNum 167 snap 16 seq_num 167) 2024-09-13T13:50:58.361 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 855788 2024-09-13T13:50:58.406 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:58.405+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:58.423 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:58.462 INFO:tasks.rados.rados.0.smithi159.stdout:1195: finishing set_chunk to oid 11 2024-09-13T13:50:58.462 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 131 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-09-13T13:50:58.462 INFO:tasks.rados.rados.0.smithi159.stdout:1197: finishing set_chunk to oid 14 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 111 (ObjNum 174 snap 16 seq_num 174) dirty exists 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: finishing write tid 1 to smithi15938319-38 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: finishing write tid 2 to smithi15938319-38 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1200: got (0) Success 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 136 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: finishing write tid 3 to smithi15938319-38 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: finishing write tid 4 to smithi15938319-38 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: finishing write tid 5 to smithi15938319-38 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: oid 38 updating version 0 to 90 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: oid 38 version 90 is already newer than 88 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: oid 38 version 90 is already newer than 89 2024-09-13T13:50:58.463 INFO:tasks.rados.rados.0.smithi159.stdout:1199: oid 38 updating version 90 to 91 2024-09-13T13:50:58.464 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 91 (ObjNum 200 snap 18 seq_num 200) dirty exists 2024-09-13T13:50:58.464 INFO:tasks.rados.rados.0.smithi159.stdout:1199: left oid 38 (ObjNum 200 snap 18 seq_num 200) 2024-09-13T13:50:58.464 INFO:tasks.rados.rados.0.smithi159.stdout:1201: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:50:58.470 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.470 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:50:58.498 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 770181 2024-09-13T13:50:58.504 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:50:58.503+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:50:58.516 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:58.605 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:58.604+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:58.627 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.627 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:50:58.706 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:50:58.704+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:50:58.725 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T13:50:58.728 INFO:tasks.rados.rados.0.smithi159.stdout:1195: done (8 left) 2024-09-13T13:50:58.728 INFO:tasks.rados.rados.0.smithi159.stdout:1196: done (7 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1197: done (6 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1198: done (5 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1199: done (4 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1200: done (3 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1201: done (2 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1202: done (1 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1203: done (0 left) 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1204: write (excl) oid 18 current snap is 19 2024-09-13T13:50:58.729 INFO:tasks.rados.rados.0.smithi159.stdout:1204: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-13T13:50:58.735 INFO:tasks.rados.rados.0.smithi159.stdout:1204: writing smithi15938319-18 from 578207 to 1128434 tid 1 2024-09-13T13:50:58.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T13:50:58.741 INFO:tasks.rados.rados.0.smithi159.stdout:1204: writing smithi15938319-18 from 1762410 to 2542144 tid 2 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1204: writing smithi15938319-18 from 3286845 to 3378207 tid 3 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1205: read oid 40 snap -1 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1205: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1206: read oid 6 snap -1 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1206: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1207: rollback oid 49 current snap is 19 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 49 to 16 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1208: read oid 38 snap -1 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1208: expect (ObjNum 200 snap 18 seq_num 200) 2024-09-13T13:50:58.747 INFO:tasks.rados.rados.0.smithi159.stdout:1209: read oid 41 snap -1 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1209: expect (ObjNum 190 snap 18 seq_num 190) 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1210: set_chunk oid 27 target oid 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1210: set_chunk oid 27 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1211: read oid 16 snap -1 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1211: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1212: write oid 21 current snap is 19 2024-09-13T13:50:58.748 INFO:tasks.rados.rados.0.smithi159.stdout:1212: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-13T13:50:58.749 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.749 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.752 INFO:tasks.rados.rados.0.smithi159.stdout:1212: writing smithi15938319-21 from 595014 to 1220490 tid 1 2024-09-13T13:50:58.759 INFO:tasks.rados.rados.0.smithi159.stdout:1212: writing smithi15938319-21 from 1820892 to 2344284 tid 2 2024-09-13T13:50:58.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1212: writing smithi15938319-21 from 2797915 to 3394391 tid 3 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1213: tier_promote oid 22 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1210: finishing set_chunk to oid 27 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1210: got expected ENOENT (src dne) 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1204: finishing write tid 1 to smithi15938319-18 2024-09-13T13:50:58.766 INFO:tasks.rados.rados.0.smithi159.stdout:1204: finishing write tid 2 to smithi15938319-18 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1204: finishing write tid 3 to smithi15938319-18 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1210: done (9 left) 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1214: tier_promote oid 12 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1207: finishing rollback tid 0 to smithi15938319-49 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1207: finishing rollback tid 1 to smithi15938319-49 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1207: finishing rollback tid 2 to smithi15938319-49 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 41 (ObjNum 173 snap 16 seq_num 173) dirty exists 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1212: finishing write tid 1 to smithi15938319-21 2024-09-13T13:50:58.767 INFO:tasks.rados.rados.0.smithi159.stdout:1208: expect (ObjNum 200 snap 18 seq_num 200) 2024-09-13T13:50:58.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:50:58.847 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:50:58.845+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:50:58.908 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:58.906+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:58.913 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:58.913 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:58.923 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:50:58.955 INFO:tasks.rados.rados.0.smithi159.stdout:1212: finishing write tid 2 to smithi15938319-21 2024-09-13T13:50:58.955 INFO:tasks.rados.rados.0.smithi159.stdout:1212: finishing write tid 3 to smithi15938319-21 2024-09-13T13:50:58.955 INFO:tasks.rados.rados.0.smithi159.stdout:1206: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:50:58.964 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 691800 2024-09-13T13:50:59.009 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:50:59.007+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:50:59.033 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.033 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:50:59.109 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:59.108+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:59.110 INFO:tasks.rados.rados.0.smithi159.stdout:1212: finishing write tid 4 to smithi15938319-21 2024-09-13T13:50:59.110 INFO:tasks.rados.rados.0.smithi159.stdout:1212: finishing write tid 5 to smithi15938319-21 2024-09-13T13:50:59.110 INFO:tasks.rados.rados.0.smithi159.stdout:1212: oid 21 updating version 0 to 90 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1212: oid 21 version 90 is already newer than 88 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1212: oid 21 version 90 is already newer than 87 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1212: oid 21 version 90 is already newer than 89 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 90 (ObjNum 202 snap 19 seq_num 202) dirty exists 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1212: left oid 21 (ObjNum 202 snap 19 seq_num 202) 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1213: got (0) Success 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 18 (ObjNum 153 snap 12 seq_num 153) dirty exists 2024-09-13T13:50:59.111 INFO:tasks.rados.rados.0.smithi159.stdout:1211: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:50:59.144 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.144 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:50:59.185 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:59.210 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:50:59.208+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:50:59.233 INFO:tasks.rados.rados.0.smithi159.stdout:1205: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:50:59.312 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:50:59.310+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:50:59.316 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:59.366 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.366 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.369 INFO:tasks.rados.rados.0.smithi159.stdout:1209: expect (ObjNum 190 snap 18 seq_num 190) 2024-09-13T13:50:59.369 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:50:59.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:50:59.415 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:59.414+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:59.416 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 233978 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1214: got (0) Success 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 84 (ObjNum 152 snap 12 seq_num 152) dirty exists 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1205: done (9 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1206: done (8 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1207: done (7 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1208: done (6 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1209: done (5 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1211: done (4 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1212: done (3 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1213: done (2 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1214: done (1 left) 2024-09-13T13:50:59.509 INFO:tasks.rados.rados.0.smithi159.stdout:1215: snap_create 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: finishing write tid 4 to smithi15938319-18 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: finishing write tid 5 to smithi15938319-18 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: oid 18 updating version 0 to 154 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: oid 18 version 154 is already newer than 153 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: oid 18 version 154 is already newer than 151 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: oid 18 version 154 is already newer than 152 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 154 (ObjNum 201 snap 19 seq_num 201) dirty exists 2024-09-13T13:50:59.510 INFO:tasks.rados.rados.0.smithi159.stdout:1204: left oid 18 (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:50:59.513 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:50:59.513+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:50:59.516 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.516 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:50:59.614 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:50:59.613+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:50:59.626 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.626 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.636 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:50:59.714 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:50:59.713+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:50:59.778 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T13:50:59.779 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1204: done (1 left) 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1215: done (0 left) 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1216: rollback oid 41 current snap is 20 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 41 to 13 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1217: read oid 21 snap -1 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1217: expect (ObjNum 202 snap 19 seq_num 202) 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1218: read oid 23 snap -1 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1218: expect (ObjNum 163 snap 13 seq_num 163) 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1219: tier_flush oid 47 2024-09-13T13:50:59.783 INFO:tasks.rados.rados.0.smithi159.stdout:1219: tier_flush oid 47 snap -1 2024-09-13T13:50:59.784 INFO:tasks.rados.rados.0.smithi159.stdout:1220: write (excl) oid 7 current snap is 20 2024-09-13T13:50:59.784 INFO:tasks.rados.rados.0.smithi159.stdout:1220: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-13T13:50:59.790 INFO:tasks.rados.rados.0.smithi159.stdout:1220: writing smithi15938319-7 from 611821 to 1312546 tid 1 2024-09-13T13:50:59.797 INFO:tasks.rados.rados.0.smithi159.stdout:1220: writing smithi15938319-7 from 1879374 to 2662777 tid 2 2024-09-13T13:50:59.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T13:50:59.801 INFO:tasks.rados.rados.0.smithi159.stdout:1220: writing smithi15938319-7 from 3341691 to 3411821 tid 3 2024-09-13T13:50:59.802 INFO:tasks.rados.rados.0.smithi159.stdout:1221: read oid 3 snap 16 2024-09-13T13:50:59.802 INFO:tasks.rados.rados.0.smithi159.stdout:1221: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:50:59.802 INFO:tasks.rados.rados.0.smithi159.stdout:1222: read oid 1 snap 16 2024-09-13T13:50:59.802 INFO:tasks.rados.rados.0.smithi159.stdout:1222: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:50:59.802 INFO:tasks.rados.rados.0.smithi159.stdout:1223: snap_remove snap 6 2024-09-13T13:50:59.815 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:50:59.814+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:50:59.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:50:59.852 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.852 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:50:59.916 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:50:59.914+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:50:59.963 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:50:59.964 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:50:59.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:50:59.984 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:50:59.984 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:00.006 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:00.006 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:00.016 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:00.014+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:00.028 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:00.028 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:00.049 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:00.069 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:00.069 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:00.090 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:00.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:00.112 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:00.112 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:00.117 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:00.115+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:00.118 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:00.118 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:00.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:00.132 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:00.133 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:00.152 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:00.153 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:00.173 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:00.173 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:00.195 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:00.195 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:00.216 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:00.217 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:00.218 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:00.216+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:00.228 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:00.227+0000 7ffaeece8640 -1 osd.0 82 handle_osd_map injecting map crc failure 2024-09-13T13:51:00.232 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:00.232 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:00.237 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:00.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:00.241 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:00.258 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:00.258 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:00.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:00.257+0000 7f77af7fe640 0 -- 172.21.15.27:0/3101819816 >> v2:172.21.15.122:6808/1255553857 conn(0x7f77b0221ba0 msgr2=0x7f77b0223f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:00.319 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:00.318+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:00.386 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:00.386 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:00.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:00.386 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.617695574000003, 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:51:00.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:51:00.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:51:00.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:51:00.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.616718376000001, 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.394 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:51:00.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.399 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.614963711000001, 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.400 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:51:00.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.614571497999997, 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.406 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:51:00.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:51:00.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.614534171000003, 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:51:00.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.418 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.609735817999997, 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:00.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:51:00.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.594100593, 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.426 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:51:00.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.431 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.593994242999997, 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:00.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:51:00.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.586060473000003, 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.438 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:51:00.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:51:00.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.444 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.567797728999999, 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:51:00.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.450 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.567671914999998, 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:00.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:51:00.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:51:00.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.456 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.567545125000002, 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.457 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:51:00.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:51:00.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.486346701999999, 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.463 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:00.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:51:00.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:51:00.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.486245251, 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:51:00.469 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:51:00.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:51:00.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:00.475 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.924892485000001, 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:51:00.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:51:00.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.481 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.919040942000002, 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:00.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.642231305999999, 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:51:00.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.490 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.657163490999999, 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.491 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:00.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:51:00.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.651142216, 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:00.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:51:00.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.503 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2394 3.1 3:93ff5947:::3634ac6462bf9cb96fe06c81293bb3c6c0e95892f99c8f55c0a14944e78c533e:head [write 0~1120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.6057543590000001, 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012999503000000001, 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2394 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.504 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778747+0000", 2024-09-13T13:51:00.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.886e-06 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780479+0000", 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017320300000000001 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780496+0000", 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7070000000000001e-05 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780522+0000", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6588999999999999e-05 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780613+0000", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0382000000000003e-05 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780619+0000", 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.353e-06 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786863+0000", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062442749999999997 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786980+0000", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116236 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.787126+0000", 2024-09-13T13:51:00.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014658299999999999 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791421+0000", 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042944840000000003 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791672+0000", 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025098399999999999 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:00.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791713+0000", 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1306000000000001e-05 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791744+0000", 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1325000000000002e-05 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:00.510 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:00.511 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:00.512 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:00.512 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:00.513 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:00.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:00.516 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:00.510+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:00.613 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:00.612+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:00.630 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:00.631 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:00.631 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:00.639 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:00.724 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:00.723+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:00.784 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:00.785 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:00.788 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1223: done (7 left) 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1224: read oid 12 snap -1 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1224: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1225: write oid 4 current snap is 20 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1216: finishing rollback tid 0 to smithi15938319-41 2024-09-13T13:51:00.791 INFO:tasks.rados.rados.0.smithi159.stdout:1225: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-13T13:51:00.794 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:00.799 INFO:tasks.rados.rados.0.smithi159.stdout:1225: writing smithi15938319-4 from 628628 to 1404602 tid 1 2024-09-13T13:51:00.803 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:00.802+0000 7f1a506ea640 -1 osd.13 83 handle_osd_map injecting map crc failure 2024-09-13T13:51:00.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T13:51:00.806 INFO:tasks.rados.rados.0.smithi159.stdout:1225: writing smithi15938319-4 from 1937856 to 2464917 tid 2 2024-09-13T13:51:00.811 INFO:tasks.rados.rados.0.smithi159.stdout:1225: writing smithi15938319-4 from 3252761 to 3428628 tid 3 2024-09-13T13:51:00.811 INFO:tasks.rados.rados.0.smithi159.stdout:1226: read oid 11 snap -1 2024-09-13T13:51:00.811 INFO:tasks.rados.rados.0.smithi159.stdout:1226: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:51:00.811 INFO:tasks.rados.rados.0.smithi159.stdout:1227: snap_create 2024-09-13T13:51:00.811 INFO:tasks.rados.rados.0.smithi159.stdout:1217: expect (ObjNum 202 snap 19 seq_num 202) 2024-09-13T13:51:00.814 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:00.813+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:00.818 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 872306 2024-09-13T13:51:00.891 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1744151 2024-09-13T13:51:00.915 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:00.914+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:00.939 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.10.22:7 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.188553716000001, 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072188460000000001, 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749185+0000", 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4400000000000001e-07 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749188+0000", 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.644e-06 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749258+0000", 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9744000000000002e-05 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.749263+0000", 2024-09-13T13:51:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6339999999999998e-06 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752867+0000", 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036041469999999998 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.752952+0000", 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4035000000000004e-05 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756404+0000", 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034522979999999999 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.039573199000003, 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:51:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:51:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:51:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:265 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.026293891999998, 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082114400000000004, 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 265 2024-09-13T13:51:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911447+0000", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911457+0000", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0433e-05 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911476+0000", 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8612000000000001e-05 2024-09-13T13:51:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911486+0000", 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911546+0000", 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9877000000000003e-05 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911627+0000", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1546999999999997e-05 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917585+0000", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059573229999999996 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918841+0000", 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001255969 2024-09-13T13:51:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919557+0000", 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000716079 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919613+0000", 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5881000000000003e-05 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919656+0000", 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3643999999999998e-05 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.023945584000003, 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:51:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:51:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.015233909000003, 2024-09-13T13:51:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:51:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:51:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:51:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 47.167687946000001, 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:51:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:51:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:366 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 46.862818206, 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079177420000000002, 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 366 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074921+0000", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074925+0000", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8010000000000001e-06 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1880000000000002e-06 2024-09-13T13:51:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074939+0000", 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075137+0000", 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197927 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075206+0000", 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8806999999999994e-05 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.078002+0000", 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002796078 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.081901+0000", 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003899713 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082743+0000", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084112100000000001 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082793+0000", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0359999999999999e-05 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082838+0000", 2024-09-13T13:51:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5321000000000002e-05 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:378 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 46.861776188999997, 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071838889999999997, 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 378 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075962+0000", 2024-09-13T13:51:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075964+0000", 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0160000000000001e-06 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075970+0000", 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9090000000000003e-06 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075981+0000", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1042e-05 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075990+0000", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2539999999999992e-06 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076042+0000", 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1999000000000001e-05 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.076116+0000", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4060000000000006e-05 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.079664+0000", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035479420000000001 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082307+0000", 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002643472 2024-09-13T13:51:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083060+0000", 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075247599999999997 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083109+0000", 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9387e-05 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.083146+0000", 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7332000000000001e-05 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1038 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 37.157714913, 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078733310000000008, 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1038 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780025+0000", 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.238e-06 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780029+0000", 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1470000000000003e-06 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780040+0000", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0597e-05 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780049+0000", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3260000000000002e-06 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780132+0000", 2024-09-13T13:51:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3185999999999999e-05 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780189+0000", 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7234000000000001e-05 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.787897+0000", 2024-09-13T13:51:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077076030000000004 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.190947758999997, 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing(4.f epoch 53/34 rep_tid 118 entries 53'57 (0'0) error 4:f883ee18:::smithi15938319-40:head by client.4571.0:1290 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 51'54)", 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.188111753000001, 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084851660000000006, 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749628+0000", 2024-09-13T13:51:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0400000000000005e-07 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749637+0000", 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749672+0000", 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4409999999999998e-05 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749685+0000", 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2754e-05 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754728+0000", 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050430300000000004 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758112+0000", 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033844600000000002 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.97627748, 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.974475854000001, 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:51:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:51:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1802 4.1 4:8bb695d2:::smithi15938319-48:c [read 3113149~634779,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.695605104999999, 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083353070000000001, 2024-09-13T13:51:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1802 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242135+0000", 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.48e-06 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242142+0000", 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6000000000000003e-06 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242164+0000", 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2356000000000001e-05 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242173+0000", 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3279999999999995e-06 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242337+0000", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016404000000000001 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242349+0000", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1440999999999999e-05 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245783+0000", 2024-09-13T13:51:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034342190000000001 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245937+0000", 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250469+0000", 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045318429999999998 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.695552287000002, 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:51:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:51:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.695528928000002, 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:51:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.14.0:137 4.b e64/34 4:def5de2b:::smithi15938319-3:head v 64'70, mlcod=61'68)", 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.677451675, 2024-09-13T13:51:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0080519279999999999, 2024-09-13T13:51:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8000000000000002e-07 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260942+0000", 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065496 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260991+0000", 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8848999999999999e-05 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261001+0000", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8749999999999995e-06 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261068+0000", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7076999999999995e-05 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261091+0000", 2024-09-13T13:51:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2648999999999999e-05 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.268297+0000", 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072060279999999997 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.268339+0000", 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2209999999999997e-05 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1856 4.1 4:8e4bdd2a:::smithi15938319-10:head [sparse-read 0~1659743] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.877030+0000", 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.060708205000001, 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073945599999999997, 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1856 2024-09-13T13:51:01.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877030+0000", 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877030+0000", 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877031+0000", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877034+0000", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4510000000000001e-06 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877044+0000", 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4239999999999999e-06 2024-09-13T13:51:01.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.877049+0000", 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0180000000000001e-06 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.883876+0000", 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0068276789999999997 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.883930+0000", 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3232999999999997e-05 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.884425+0000", 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049537599999999995 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:01.039 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 9.1970840910000007, 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:51:01.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.14.0:202 4.b e75/34 4:def5de2b:::smithi15938319-3:head v 75'104, mlcod=74'101)", 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.745258+0000", 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 9.1924806920000002, 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075299499999999997, 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.047 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745258+0000", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745258+0000", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745259+0000", 2024-09-13T13:51:01.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.094e-06 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745908+0000", 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064862199999999996 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745946+0000", 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8059000000000003e-05 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.745955+0000", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5850000000000006e-06 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.746020+0000", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4443000000000006e-05 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.746041+0000", 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1637999999999998e-05 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.752743+0000", 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067012340000000004 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.752788+0000", 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5275000000000003e-05 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.051 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3095724 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:1219: got (16) Device or resource busy 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 47 v 44 (ObjNum 115 snap 8 seq_num 115) unflushed 13 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:1216: finishing rollback tid 1 to smithi15938319-41 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:1216: finishing rollback tid 2 to smithi15938319-41 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 136 (ObjNum 119 snap 8 seq_num 119) dirty exists 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:1218: expect (ObjNum 163 snap 13 seq_num 163) 2024-09-13T13:51:01.053 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:01.054 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:01.051+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:01.054 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:01.058 INFO:tasks.rados.rados.0.smithi159.stdout:1222: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:01.153 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:01.152+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: finishing write tid 5 to smithi15938319-7 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: oid 7 updating version 0 to 127 2024-09-13T13:51:01.155 INFO:tasks.rados.rados.0.smithi159.stdout:1220: oid 7 updating version 127 to 128 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:1220: oid 7 version 128 is already newer than 125 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:1220: oid 7 version 128 is already newer than 126 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 128 (ObjNum 203 snap 20 seq_num 203) dirty exists 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:1220: left oid 7 (ObjNum 203 snap 20 seq_num 203) 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:1221: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:51:01.156 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:01.157 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.158 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:01.158 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:01.158 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.166 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:01.188 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:01.210 INFO:tasks.rados.rados.0.smithi159.stdout:1224: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:01.210 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:01.254 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:01.253+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:01.269 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:01.270 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:01.351 INFO:tasks.rados.rados.0.smithi159.stdout:1225: finishing write tid 1 to smithi15938319-4 2024-09-13T13:51:01.351 INFO:tasks.rados.rados.0.smithi159.stdout:1225: finishing write tid 2 to smithi15938319-4 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: finishing write tid 3 to smithi15938319-4 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: finishing write tid 4 to smithi15938319-4 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: finishing write tid 5 to smithi15938319-4 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: oid 4 updating version 0 to 133 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: oid 4 updating version 133 to 134 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: oid 4 updating version 134 to 136 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: oid 4 version 136 is already newer than 135 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 136 (ObjNum 204 snap 20 seq_num 204) dirty exists 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1225: left oid 4 (ObjNum 204 snap 20 seq_num 204) 2024-09-13T13:51:01.352 INFO:tasks.rados.rados.0.smithi159.stdout:1226: expect (ObjNum 65 snap 3 seq_num 65) 2024-09-13T13:51:01.355 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:01.355+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:01.373 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2749703 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.383 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 72.839682429000007, 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:51:01.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:51:01.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.785356539999995, 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.390 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:51:01.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.771978149999995, 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:51:01.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.958645212999997, 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:51:01.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:51:01.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:01.404 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153233206000003, 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:51:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153196033999997, 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:51:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:51:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153175306000001, 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:51:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:51:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153079005000002, 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153056360999997, 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:51:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:51:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:51:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.153030330999997, 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:51:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.013529464000001, 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.013422517000002, 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:51:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:51:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.013393258000001, 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:51:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:51:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.007028855000001, 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:51:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:51:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:51:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 33.002068661000003, 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:51:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:51:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:51:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1279 4.4 4:22f57a3c:::smithi15938319-42:head [write 2555481~425187] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.997224529, 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.37714365399999999, 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1279 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384128+0000", 2024-09-13T13:51:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384129+0000", 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2399999999999996e-07 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384523+0000", 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039437499999999998 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384535+0000", 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1626e-05 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384541+0000", 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4699999999999999e-06 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1349e-05 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384589+0000", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3249999999999996e-06 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728981+0000", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34439230599999998 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729015+0000", 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4149000000000001e-05 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740625+0000", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011609895 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740676+0000", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.13e-05 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754660+0000", 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013983423 2024-09-13T13:51:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754774+0000", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114554 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754954+0000", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017924600000000001 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759913+0000", 2024-09-13T13:51:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049594189999999996 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761144+0000", 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001231174 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761182+0000", 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8235999999999998e-05 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761271+0000", 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8882999999999996e-05 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1280 4.4 4:22f57a3c:::smithi15938319-42:head [setxattr _header (56) in=7b,truncate 3512630] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.996797532000002, 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379144494, 2024-09-13T13:51:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1280 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384555+0000", 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.99e-07 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384559+0000", 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384564+0000", 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0610000000000003e-06 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384569+0000", 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384617+0000", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8479000000000001e-05 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384623+0000", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.677e-06 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729055+0000", 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34443185999999998 2024-09-13T13:51:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729104+0000", 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9335000000000001e-05 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740720+0000", 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011616191 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740766+0000", 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5667000000000002e-05 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759391+0000", 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018624900999999999 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759450+0000", 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8891000000000002e-05 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759563+0000", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011324600000000001 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.761528+0000", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019654220000000001 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763564+0000", 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020357410000000002 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763611+0000", 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6873000000000003e-05 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763699+0000", 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8288000000000006e-05 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1281 4.4 4:22f57a3c:::smithi15938319-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.996770154000004, 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.379374818, 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1281 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384582+0000", 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1899999999999998e-07 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384585+0000", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.334e-06 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384591+0000", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8259999999999998e-06 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384596+0000", 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6439999999999997e-06 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384640+0000", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3217999999999999e-05 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.384643+0000", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6330000000000002e-06 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729130+0000", 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34448669500000001 2024-09-13T13:51:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729160+0000", 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0035999999999998e-05 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740807+0000", 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011646844 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740902+0000", 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4952999999999997e-05 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760235+0000", 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019333605 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760386+0000", 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015056100000000001 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763739+0000", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033526620000000002 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763822+0000", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3255000000000002e-05 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763957+0000", 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013513300000000001 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.060078944000001, 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:51:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.060039805999999, 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:51:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:51:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.510 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.511 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2044450 2024-09-13T13:51:01.512 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:01.512 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:01.511+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:01.612 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:01.611+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:01.654 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.654 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:01.655 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:01.655 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:01.713 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:01.712+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:01.766 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.766 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:01.766 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:01.766 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:01.766 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:01.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:51:01.789 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:01.788+0000 7fc16c184640 -1 osd.11 84 handle_osd_map injecting map crc failure 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1216: done (10 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1217: done (9 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1218: done (8 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1219: done (7 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1220: done (6 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1221: done (5 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1222: done (4 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1224: done (3 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1225: done (2 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1226: done (1 left) 2024-09-13T13:51:01.793 INFO:tasks.rados.rados.0.smithi159.stdout:1227: done (0 left) 2024-09-13T13:51:01.794 INFO:tasks.rados.rados.0.smithi159.stdout:1228: write oid 47 current snap is 21 2024-09-13T13:51:01.794 INFO:tasks.rados.rados.0.smithi159.stdout:1228: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-13T13:51:01.795 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:01.794+0000 7f2282f0c640 -1 osd.10 84 handle_osd_map injecting map crc failure 2024-09-13T13:51:01.798 INFO:tasks.rados.rados.0.smithi159.stdout:1228: writing smithi15938319-47 from 645435 to 1096658 tid 1 2024-09-13T13:51:01.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:51:01.806 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-09-13T13:51:01.806 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-09-13T13:51:01.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T13:51:01.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:01.806 INFO:tasks.rados.rados.0.smithi159.stdout:1228: writing smithi15938319-47 from 1879985 to 2667057 tid 2 2024-09-13T13:51:01.811 INFO:tasks.rados.rados.0.smithi159.stdout:1228: writing smithi15938319-47 from 3280184 to 3445435 tid 3 2024-09-13T13:51:01.811 INFO:tasks.rados.rados.0.smithi159.stdout:1229: write (excl) oid 30 current snap is 21 2024-09-13T13:51:01.811 INFO:tasks.rados.rados.0.smithi159.stdout:1229: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-13T13:51:01.814 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:01.813+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:01.817 INFO:tasks.rados.rados.0.smithi159.stdout:1229: writing smithi15938319-30 from 662242 to 1305067 tid 1 2024-09-13T13:51:01.824 INFO:tasks.rados.rados.0.smithi159.stdout:1229: writing smithi15938319-30 from 2054820 to 2585550 tid 2 2024-09-13T13:51:01.829 INFO:tasks.rados.rados.0.smithi159.stdout:1229: writing smithi15938319-30 from 3307607 to 3462242 tid 3 2024-09-13T13:51:01.829 INFO:tasks.rados.rados.0.smithi159.stdout:1230: write (excl) oid 13 current snap is 21 2024-09-13T13:51:01.829 INFO:tasks.rados.rados.0.smithi159.stdout:1230: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-13T13:51:01.836 INFO:tasks.rados.rados.0.smithi159.stdout:1230: writing smithi15938319-13 from 679049 to 1397123 tid 1 2024-09-13T13:51:01.844 INFO:tasks.rados.rados.0.smithi159.stdout:1230: writing smithi15938319-13 from 2113302 to 2787690 tid 2 2024-09-13T13:51:01.848 INFO:tasks.rados.rados.0.smithi159.stdout:1230: writing smithi15938319-13 from 3335030 to 3479049 tid 3 2024-09-13T13:51:01.848 INFO:tasks.rados.rados.0.smithi159.stdout:1231: read oid 27 snap -1 2024-09-13T13:51:01.848 INFO:tasks.rados.rados.0.smithi159.stdout:1231: expect deleted 2024-09-13T13:51:01.849 INFO:tasks.rados.rados.0.smithi159.stdout:1232: set_chunk oid 16 target oid 2024-09-13T13:51:01.858 INFO:tasks.rados.rados.0.smithi159.stdout:1232: set_chunk oid 16 offset: 1413120 length: 762368 target oid acb594674c64d5964231fdc29d44c224f52158f0d2fa8ffc034a43dc5bff85c8 offset: 0 snap -1 2024-09-13T13:51:01.858 INFO:tasks.rados.rados.0.smithi159.stdout:1233: rollback oid 31 current snap is 21 2024-09-13T13:51:01.858 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 31 to 13 2024-09-13T13:51:01.858 INFO:tasks.rados.rados.0.smithi159.stdout:1229: finishing write tid 1 to smithi15938319-30 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: finishing write tid 2 to smithi15938319-30 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: finishing write tid 3 to smithi15938319-30 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: finishing write tid 4 to smithi15938319-30 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: finishing write tid 5 to smithi15938319-30 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: oid 30 updating version 0 to 139 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: oid 30 updating version 139 to 140 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: oid 30 version 140 is already newer than 138 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: oid 30 updating version 140 to 141 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 141 (ObjNum 206 snap 21 seq_num 206) dirty exists 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: left oid 30 (ObjNum 206 snap 21 seq_num 206) 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1230: finishing write tid 1 to smithi15938319-13 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1229: done (5 left) 2024-09-13T13:51:01.859 INFO:tasks.rados.rados.0.smithi159.stdout:1234: delete oid 30 current snap is 21 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: finishing write tid 2 to smithi15938319-13 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: finishing write tid 3 to smithi15938319-13 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: finishing write tid 4 to smithi15938319-13 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: finishing write tid 5 to smithi15938319-13 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: oid 13 updating version 0 to 81 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: oid 13 updating version 81 to 82 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: oid 13 updating version 82 to 84 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: oid 13 version 84 is already newer than 83 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 84 (ObjNum 207 snap 21 seq_num 207) dirty exists 2024-09-13T13:51:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:1230: left oid 13 (ObjNum 207 snap 21 seq_num 207) 2024-09-13T13:51:01.862 INFO:tasks.rados.rados.0.smithi159.stdout:1230: done (5 left) 2024-09-13T13:51:01.862 INFO:tasks.rados.rados.0.smithi159.stdout:1231: done (4 left) 2024-09-13T13:51:01.862 INFO:tasks.rados.rados.0.smithi159.stdout:1234: done (3 left) 2024-09-13T13:51:01.863 INFO:tasks.rados.rados.0.smithi159.stdout:1235: tier_evict oid 10 2024-09-13T13:51:01.863 INFO:tasks.rados.rados.0.smithi159.stdout:1235: tier_evict oid 10 snap -1 2024-09-13T13:51:01.863 INFO:tasks.rados.rados.0.smithi159.stdout:1236: read oid 28 snap 16 2024-09-13T13:51:01.863 INFO:tasks.rados.rados.0.smithi159.stdout:1236: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:51:01.863 INFO:tasks.rados.rados.0.smithi159.stdout:1237: set_chunk oid 40 target oid 2024-09-13T13:51:01.869 INFO:tasks.rados.rados.0.smithi159.stdout:1237: set_chunk oid 40 offset: 45568 length: 693760 target oid 0bf2f9f83cea0d9d2a8ac585000560355b42bf3d915d2494b4c6296927430b1d offset: 0 snap -1 2024-09-13T13:51:01.869 INFO:tasks.rados.rados.0.smithi159.stdout:1238: write oid 19 current snap is 21 2024-09-13T13:51:01.869 INFO:tasks.rados.rados.0.smithi159.stdout:1235: got (22) Invalid argument 2024-09-13T13:51:01.869 INFO:tasks.rados.rados.0.smithi159.stdout:1238: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-13T13:51:01.875 INFO:tasks.rados.rados.0.smithi159.stdout:1238: writing smithi15938319-19 from 712663 to 1181235 tid 1 2024-09-13T13:51:01.882 INFO:tasks.rados.rados.0.smithi159.stdout:1238: writing smithi15938319-19 from 1713913 to 2391970 tid 2 2024-09-13T13:51:01.886 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:01.886 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.147681962, 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.887 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:51:01.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.142441339999998, 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.893 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:51:01.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.142390026000001, 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:51:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:51:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.142317829, 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:51:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:51:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:51:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.142278824000002, 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:51:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:51:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.142214412000001, 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:51:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.116864315999997, 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:51:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:51:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:51:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.086303491999999, 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:51:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:51:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:51:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.064976905000002, 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:51:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.986760627999999, 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:51:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:51:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642788968000001, 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:51:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:51:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642730916000001, 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:51:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:51:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:51:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642704956999999, 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:51:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:51:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642590541000001, 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:51:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:51:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:51:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642559649999999, 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:51:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:51:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:51:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.642523066999999, 2024-09-13T13:51:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:51:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.562728196999998, 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.188067779000001, 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:51:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:51:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.064876256, 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:51:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:51:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.0568559580000001, 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:51:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.005 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:02.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:02.008 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:02.005+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:02.008 INFO:tasks.rados.rados.0.smithi159.stdout:1238: writing smithi15938319-19 from 2873523 to 3465256 tid 3 2024-09-13T13:51:02.008 INFO:tasks.rados.rados.0.smithi159.stdout:1235: done (6 left) 2024-09-13T13:51:02.008 INFO:tasks.rados.rados.0.smithi159.stdout:1239: snap_create 2024-09-13T13:51:02.008 INFO:tasks.rados.rados.0.smithi159.stdout:1232: finishing set_chunk to oid 16 2024-09-13T13:51:02.008 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 169 (ObjNum 199 snap 18 seq_num 199) dirty exists 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1233: finishing rollback tid 0 to smithi15938319-31 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1233: finishing rollback tid 1 to smithi15938319-31 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1233: finishing rollback tid 2 to smithi15938319-31 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 144 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1237: finishing set_chunk to oid 40 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 121 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: finishing write tid 1 to smithi15938319-19 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: finishing write tid 2 to smithi15938319-19 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: finishing write tid 3 to smithi15938319-19 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: finishing write tid 4 to smithi15938319-19 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: finishing write tid 5 to smithi15938319-19 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: oid 19 updating version 0 to 92 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: oid 19 updating version 92 to 94 2024-09-13T13:51:02.009 INFO:tasks.rados.rados.0.smithi159.stdout:1238: oid 19 version 94 is already newer than 93 2024-09-13T13:51:02.010 INFO:tasks.rados.rados.0.smithi159.stdout:1238: oid 19 version 94 is already newer than 91 2024-09-13T13:51:02.010 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 94 (ObjNum 209 snap 21 seq_num 209) dirty exists 2024-09-13T13:51:02.010 INFO:tasks.rados.rados.0.smithi159.stdout:1238: left oid 19 (ObjNum 209 snap 21 seq_num 209) 2024-09-13T13:51:02.107 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:02.107+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:02.151 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:02.151 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:02.151 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:02.151 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:02.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:02.173 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:51:02.174 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":85,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:51:01.787684+0000","last_up_change":"2024-09-13T13:49:47.205190+0000","last_in_change":"2024-09-13T13:50:28.178948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"reef","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T13:50:03.138857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-09-13T13:50:04.433048+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":22,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":2.630000114440918,"raw_score_stable":2.630000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":277461260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":277461260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":277461260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":277461260}]},"public_addr":"172.21.15.27:6812/277461260","cluster_addr":"172.21.15.27:6813/277461260","heartbeat_back_addr":"172.21.15.27:6815/277461260","heartbeat_front_addr":"172.21.15.27:6814/277461260","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:48.370677+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.987145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:57.703681+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:59.850472+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:50.576723+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:47.412003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:53.098767+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.913247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.205344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.924694+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:49:44.492471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:53.478425+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:53.450523+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:53.868245+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:46.926872+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:50:46.556068+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":11,"to":15}]}],"pg_upmap_primaries":[{"pgid":"2.2","primary_osd":9}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":8,"length":1},{"begin":11,"length":1},{"begin":14,"length":1},{"begin":16,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:02.187 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T13:51:02.209 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:02.208+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:02.302 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:02.303 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:02.303 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:02.310 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:02.309+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:02.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:02.411 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:02.411+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:02.456 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 72.226858237000002, 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:51:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:51:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.722120054999998, 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:51:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.722070743000003, 2024-09-13T13:51:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:51:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.722055038999997, 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:51:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.722027363000002, 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:51:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:51:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:51:02.488 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.686809765, 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:51:02.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:51:02.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.679740783, 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.495 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:51:02.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:51:02.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.645202320999999, 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.501 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:51:02.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:51:02.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:51:02.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.637572478999999, 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:51:02.507 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:02.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:51:02.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:51:02.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.513 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.634196107000001, 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:51:02.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.519 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.531336826999997, 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:51:02.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.525 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 45.320313286999998, 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 45.308798004000003, 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:51:02.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:02.534 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.207489536000001, 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:51:02.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:51:02.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.207425187999998, 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.540 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:51:02.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:51:02.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.207397126, 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.546 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:51:02.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.551 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.205715236, 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:51:02.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:02.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:51:02.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:51:02.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1811 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:02.559 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.198662215999999, 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030548815999999999, 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1811 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:02.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255828+0000", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255835+0000", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2119999999999997e-06 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:51:02.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256027+0000", 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018704600000000001 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256075+0000", 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260766+0000", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046910119999999996 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260793+0000", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6594999999999999e-05 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280369+0000", 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019576296 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280418+0000", 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9221999999999997e-05 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286090+0000", 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005671873 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286195+0000", 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010464299999999999 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286374+0000", 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179118 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:51:02.566 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.766187028000001, 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:51:02.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.6375512730000001, 2024-09-13T13:51:02.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:02.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:51:02.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:51:02.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:02.576 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:02.577 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:02.578 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:02.575+0000 7f586a715640 -1 osd.9 84 handle_osd_map injecting map crc failure 2024-09-13T13:51:02.579 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:02.579 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:02.577+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:02.679 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:02.678+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:02.692 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:02.693 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:02.693 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:02.693 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:02.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:02.715 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:02.716 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:02.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:02.738 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:02.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:02.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:02.779 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:02.777+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:02.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:02.781 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:02.801 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:02.805 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:02.805 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:02.808 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:02.807+0000 7f2284f10640 -1 osd.10 85 handle_osd_map injecting map crc failure 2024-09-13T13:51:02.810 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:02.809+0000 7f8f5ac65640 -1 osd.6 85 handle_osd_map injecting map crc failure 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1232: done (6 left) 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1233: done (5 left) 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1237: done (4 left) 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1238: done (3 left) 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1239: done (2 left) 2024-09-13T13:51:02.810 INFO:tasks.rados.rados.0.smithi159.stdout:1240: write (excl) oid 44 current snap is 22 2024-09-13T13:51:02.811 INFO:tasks.rados.rados.0.smithi159.stdout:1240: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-13T13:51:02.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:51:02.817 INFO:tasks.rados.rados.0.smithi159.stdout:1240: writing smithi15938319-44 from 729470 to 1273291 tid 1 2024-09-13T13:51:02.822 INFO:tasks.rados.rados.0.smithi159.stdout:1240: writing smithi15938319-44 from 1772395 to 2310463 tid 2 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1240: writing smithi15938319-44 from 2900946 to 3303951 tid 3 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1241: read oid 36 snap 13 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1241: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1242: read oid 6 snap -1 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1242: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:51:02.829 INFO:tasks.rados.rados.0.smithi159.stdout:1243: read oid 49 snap -1 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1243: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1244: read oid 18 snap -1 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1244: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1245: read oid 1 snap 21 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1245: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1246: write oid 33 current snap is 22 2024-09-13T13:51:02.830 INFO:tasks.rados.rados.0.smithi159.stdout:1246: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-13T13:51:02.831 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:02.831 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:02.836 INFO:tasks.rados.rados.0.smithi159.stdout:1246: writing smithi15938319-33 from 746277 to 1365347 tid 1 2024-09-13T13:51:02.845 INFO:tasks.rados.rados.0.smithi159.stdout:1246: writing smithi15938319-33 from 1830877 to 2512603 tid 2 2024-09-13T13:51:02.851 INFO:tasks.rados.rados.0.smithi159.stdout:1246: writing smithi15938319-33 from 2928369 to 3546277 tid 3 2024-09-13T13:51:02.851 INFO:tasks.rados.rados.0.smithi159.stdout:1247: write (excl) oid 38 current snap is 22 2024-09-13T13:51:02.851 INFO:tasks.rados.rados.0.smithi159.stdout:1247: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-13T13:51:02.858 INFO:tasks.rados.rados.0.smithi159.stdout:1247: writing smithi15938319-38 from 763084 to 1457403 tid 1 2024-09-13T13:51:02.859 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:02.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:02.864 INFO:tasks.rados.rados.0.smithi159.stdout:1247: writing smithi15938319-38 from 1889359 to 2431096 tid 2 2024-09-13T13:51:02.870 INFO:tasks.rados.rados.0.smithi159.stdout:1247: writing smithi15938319-38 from 2955792 to 3497694 tid 3 2024-09-13T13:51:02.870 INFO:tasks.rados.rados.0.smithi159.stdout:1248: snap_remove snap 19 2024-09-13T13:51:02.879 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:02.878+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:02.881 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:02.881 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:02.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:02.902 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:02.921 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:02.921 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:02.923 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:02.923 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:02.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:02.945 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:02.965 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:02.966 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:02.974 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:02.973+0000 7fe0f9e03640 -1 osd.2 85 handle_osd_map injecting map crc failure 2024-09-13T13:51:02.980 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:02.979+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:02.986 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:02.986 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:03.008 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:03.008 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:03.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:03.013+0000 7faeb933f640 0 -- 172.21.15.27:0/1597496057 >> v2:172.21.15.122:6808/1255553857 conn(0x7faeb4221a70 msgr2=0x7faeb4223ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:03.035 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:03.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:03.056 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:03.056 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:03.073 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.073 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:03.082 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:03.081+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:03.182 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:03.182+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:03.185 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.185 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.194 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:03.283 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:03.283+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:03.296 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.297 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:03.384 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:03.384+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:03.408 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.408 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.418 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:03.484 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:03.484+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:03.522 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.522 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:03.585 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:03.584+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:03.633 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.633 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.641 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:03.686 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:03.685+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:03.732 INFO:tasks.rados.rados.0.smithi159.stdout:1248: done (10 left) 2024-09-13T13:51:03.732 INFO:tasks.rados.rados.0.smithi159.stdout:1249: read oid 45 snap -1 2024-09-13T13:51:03.732 INFO:tasks.rados.rados.0.smithi159.stdout:1249: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:51:03.733 INFO:tasks.rados.rados.0.smithi159.stdout:1250: tier_evict oid 50 2024-09-13T13:51:03.733 INFO:tasks.rados.rados.0.smithi159.stdout:1250: tier_evict oid 50 snap -1 2024-09-13T13:51:03.733 INFO:tasks.rados.rados.0.smithi159.stdout:1243: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:03.746 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.746 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:03.801 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:03.800+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:03.859 INFO:tasks.rados.rados.0.smithi159.stdout:1245: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:03.860 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:03.860 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:03.869 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:03.910 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:03.909+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:03.939 INFO:tasks.rados.rados.0.smithi159.stdout:1228: finishing write tid 1 to smithi15938319-47 2024-09-13T13:51:03.940 INFO:tasks.rados.rados.0.smithi159.stdout:1228: finishing write tid 2 to smithi15938319-47 2024-09-13T13:51:03.940 INFO:tasks.rados.rados.0.smithi159.stdout:1242: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:51:03.991 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:03.991+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:03.996 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.015 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.015 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:04.060 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1017928 2024-09-13T13:51:04.093 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:04.092+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1240: finishing write tid 1 to smithi15938319-44 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1240: finishing write tid 2 to smithi15938319-44 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1240: finishing write tid 3 to smithi15938319-44 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1242: done (11 left) 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1243: done (10 left) 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1245: done (9 left) 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1251: read oid 8 snap -1 2024-09-13T13:51:04.098 INFO:tasks.rados.rados.0.smithi159.stdout:1251: expect deleted 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1252: read oid 12 snap 21 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1252: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1253: read oid 20 snap -1 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1253: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1254: delete oid 13 current snap is 22 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: finishing write tid 4 to smithi15938319-44 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: finishing write tid 5 to smithi15938319-44 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: oid 44 updating version 0 to 146 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: oid 44 version 146 is already newer than 145 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: oid 44 updating version 146 to 147 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: oid 44 version 147 is already newer than 144 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 147 (ObjNum 210 snap 22 seq_num 210) dirty exists 2024-09-13T13:51:04.099 INFO:tasks.rados.rados.0.smithi159.stdout:1240: left oid 44 (ObjNum 210 snap 22 seq_num 210) 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: finishing write tid 3 to smithi15938319-47 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: finishing write tid 4 to smithi15938319-47 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: finishing write tid 5 to smithi15938319-47 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: oid 47 updating version 0 to 102 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: oid 47 version 102 is already newer than 101 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: oid 47 version 102 is already newer than 100 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: oid 47 version 102 is already newer than 99 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 102 (ObjNum 205 snap 21 seq_num 205) dirty exists 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1228: left oid 47 (ObjNum 205 snap 21 seq_num 205) 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:1244: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:04.100 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.133 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.134 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:04.180 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.193 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:04.192+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:04.249 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.249 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:04.279 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1464044 2024-09-13T13:51:04.301 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:04.300+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: finishing write tid 1 to smithi15938319-38 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: finishing write tid 2 to smithi15938319-38 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: finishing write tid 3 to smithi15938319-38 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: finishing write tid 4 to smithi15938319-38 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: finishing write tid 5 to smithi15938319-38 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: oid 38 updating version 0 to 107 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: oid 38 version 107 is already newer than 106 2024-09-13T13:51:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1247: oid 38 version 107 is already newer than 104 2024-09-13T13:51:04.336 INFO:tasks.rados.rados.0.smithi159.stdout:1247: oid 38 version 107 is already newer than 105 2024-09-13T13:51:04.336 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 107 (ObjNum 212 snap 22 seq_num 212) dirty exists 2024-09-13T13:51:04.336 INFO:tasks.rados.rados.0.smithi159.stdout:1247: left oid 38 (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:04.336 INFO:tasks.rados.rados.0.smithi159.stdout:1250: got (22) Invalid argument 2024-09-13T13:51:04.336 INFO:tasks.rados.rados.0.smithi159.stdout:1249: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:51:04.364 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.364 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:04.383 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.414 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:04.413+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:04.481 INFO:tasks.rados.rados.0.smithi159.stdout:1236: expect (ObjNum 134 snap 11 seq_num 134) 2024-09-13T13:51:04.497 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:04.497+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:04.520 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1551452 2024-09-13T13:51:04.521 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.521 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:04.556 INFO:tasks.rados.rados.0.smithi159.stdout:1241: expect (ObjNum 133 snap 9 seq_num 133) 2024-09-13T13:51:04.557 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.601 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1161038 2024-09-13T13:51:04.601 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:04.601+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:04.617 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.636 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.636 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: finishing write tid 1 to smithi15938319-33 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: finishing write tid 2 to smithi15938319-33 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: finishing write tid 3 to smithi15938319-33 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: finishing write tid 4 to smithi15938319-33 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: finishing write tid 5 to smithi15938319-33 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: oid 33 updating version 0 to 151 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: oid 33 version 151 is already newer than 150 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: oid 33 version 151 is already newer than 149 2024-09-13T13:51:04.651 INFO:tasks.rados.rados.0.smithi159.stdout:1246: oid 33 updating version 151 to 152 2024-09-13T13:51:04.652 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 152 (ObjNum 211 snap 22 seq_num 211) dirty exists 2024-09-13T13:51:04.652 INFO:tasks.rados.rados.0.smithi159.stdout:1246: left oid 33 (ObjNum 211 snap 22 seq_num 211) 2024-09-13T13:51:04.652 INFO:tasks.rados.rados.0.smithi159.stdout:1252: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:04.652 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.702 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:04.702+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:04.741 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.751 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.751 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:04.792 INFO:tasks.rados.rados.0.smithi159.stdout:1253: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:51:04.792 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.803 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:04.803+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:04.813 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:04.811+0000 7faeac214640 -1 osd.8 87 handle_osd_map injecting map crc failure 2024-09-13T13:51:04.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:51:04.855 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:04.872 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:04.872 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:04.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:04.904 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:04.903+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:04.921 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 594294 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1228: done (12 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1236: done (11 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1240: done (10 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1241: done (9 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1244: done (8 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1246: done (7 left) 2024-09-13T13:51:04.978 INFO:tasks.rados.rados.0.smithi159.stdout:1247: done (6 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1249: done (5 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1250: done (4 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1251: done (3 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1252: done (2 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1253: done (1 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1254: done (0 left) 2024-09-13T13:51:04.979 INFO:tasks.rados.rados.0.smithi159.stdout:1255: snap_create 2024-09-13T13:51:05.005 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:05.005+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:05.031 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:05.031 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:05.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:05.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:05.106 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:05.106+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:05.149 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:05.149 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:05.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:05.207 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:05.206+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:05.219 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:05.219 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:05.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:05.242 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:05.263 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:05.263 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:05.268 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:05.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:05.285 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:05.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:05.308 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:05.307+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:05.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:05.329 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:05.349 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:05.350 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:05.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:05.371 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:05.385 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:05.385 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:05.392 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:05.392 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:05.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:05.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:05.414 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:05.414 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:05.413+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:05.436 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:05.457 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:05.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:05.478 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:05.478 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:05.500 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:05.510 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:05.509+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:05.521 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:05.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:05.520+0000 7f99b7fff640 0 -- 172.21.15.27:0/1516017886 >> v2:172.21.15.159:6805/4073091144 conn(0x7f99b8224fd0 msgr2=0x7f99b82273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 52.623843825999998, 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:51:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:51:05.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:51:05.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:51:05.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 52.623775969, 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:51:05.530 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:51:05.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:51:05.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 52.610493044000002, 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.536 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:51:05.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 51.447216376, 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.542 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:51:05.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:51:05.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:51:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1087 4.c 4:34c00adb:::smithi15938319-12:head [write 460525~690852] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:05.547 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 41.161718174000001, 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010491474000000001, 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1087 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:05.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360281+0000", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0999999999999998e-06 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364170+0000", 2024-09-13T13:51:05.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038886929999999999 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364189+0000", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9185999999999999e-05 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364198+0000", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364278+0000", 2024-09-13T13:51:05.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9839999999999995e-05 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364399+0000", 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364616+0000", 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021726100000000001 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.366707+0000", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002091341 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370646+0000", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039392430000000003 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370714+0000", 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8069000000000002e-05 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370771+0000", 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6292000000000002e-05 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293801357, 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:51:05.553 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:51:05.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:51:05.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:51:05.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293766882, 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.558 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:51:05.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:51:05.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293751896000003, 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:51:05.563 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:51:05.567 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293737847000003, 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.568 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:51:05.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293716132, 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.573 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:51:05.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:05.577 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.293684286999998, 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131987561000003, 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:51:05.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131954411000002, 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:51:05.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:05.592 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131924193000003, 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:05.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.597 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131881843999999, 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.603 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131847626999999, 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.131814356, 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:51:05.609 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:51:05.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.729531123000001, 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.618 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.7044645140000001, 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:51:05.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2587 4.c 4:34327e96:::smithi15938319-44:head [write 1772395~538068] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 2.6987015589999999, 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012151192999999999, 2024-09-13T13:51:05.626 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2587 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823297+0000", 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823897+0000", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060006600000000001 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823910+0000", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2243e-05 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823920+0000", 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0538e-05 2024-09-13T13:51:05.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825222+0000", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001302194 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825304+0000", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1953999999999995e-05 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825429+0000", 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012415399999999999 2024-09-13T13:51:05.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826790+0000", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013615050000000001 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835350+0000", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085598610000000002 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835406+0000", 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5661000000000001e-05 2024-09-13T13:51:05.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835447+0000", 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:05.632 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:05.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:05.634 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:05.633+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:05.734 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:05.734+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:05.743 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:05.743 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:05.749 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:05.749 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:05.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:05.818 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:51:05.821 INFO:tasks.rados.rados.0.smithi159.stdout:1255: done (0 left) 2024-09-13T13:51:05.821 INFO:tasks.rados.rados.0.smithi159.stdout:1256: read oid 19 snap -1 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1256: expect (ObjNum 209 snap 21 seq_num 209) 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1257: tier_flush oid 8 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1257: tier_flush oid 8 snap -1 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1258: set_chunk oid 13 target oid 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1258: set_chunk oid 13 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1259: read oid 41 snap -1 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1259: expect (ObjNum 119 snap 8 seq_num 119) 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1260: write oid 4 current snap is 23 2024-09-13T13:51:05.822 INFO:tasks.rados.rados.0.smithi159.stdout:1260: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-13T13:51:05.822 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:05.820+0000 7f26bcfe2640 -1 osd.3 88 handle_osd_map injecting map crc failure 2024-09-13T13:51:05.828 INFO:tasks.rados.rados.0.smithi159.stdout:1260: writing smithi15938319-4 from 796698 to 1357868 tid 1 2024-09-13T13:51:05.834 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:05.833+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:05.835 INFO:tasks.rados.rados.0.smithi159.stdout:1260: writing smithi15938319-4 from 2006323 to 2551729 tid 2 2024-09-13T13:51:05.841 INFO:tasks.rados.rados.0.smithi159.stdout:1260: writing smithi15938319-4 from 3126991 to 3596698 tid 3 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1261: snap_create 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1257: got (2) No such file or directory 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1257: got expected ENOENT (src dne) 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1258: finishing set_chunk to oid 13 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1258: got expected ERANGE (src dne) 2024-09-13T13:51:05.842 INFO:tasks.rados.rados.0.smithi159.stdout:1259: expect (ObjNum 119 snap 8 seq_num 119) 2024-09-13T13:51:05.844 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 575923 2024-09-13T13:51:05.863 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:05.863 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:05.863 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:05.863 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:05.863 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:05.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:05.876 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:05.898 INFO:tasks.rados.rados.0.smithi159.stdout:1256: expect (ObjNum 209 snap 21 seq_num 209) 2024-09-13T13:51:05.935 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:05.934+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.059626709, 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:51:05.977 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:05.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:05.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:51:05.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:05.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.983 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.054827179999997, 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:05.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:05.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:51:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:51:05.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.990 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 52.205479142000002, 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:05.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:51:05.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:51:05.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:05.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:05.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.900678831, 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:05.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:05.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:51:06.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.811797587000001, 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.004 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:51:06.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.701181542, 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.011 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.695321593000003, 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.019 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:51:06.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:51:06.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:51:06.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.690376952000001, 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.029 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:51:06.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:51:06.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:51:06.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:51:06.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:51:06.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.041 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.690166374, 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:06.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.690118275000003, 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.050 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:06.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.607037863999999, 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:51:06.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.606929153000003, 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:51:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.606813873, 2024-09-13T13:51:06.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:51:06.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:51:06.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1328 4.7 4:e26a40dd:::smithi15938319-25:head [write 796665~544891] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 36.025414044999998, 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085750689999999994, 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1328 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:06.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949505+0000", 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1409999999999999e-06 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950074+0000", 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056815299999999995 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950091+0000", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7385999999999999e-05 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950118+0000", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7101e-05 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950198+0000", 2024-09-13T13:51:06.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9474999999999994e-05 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950204+0000", 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4659999999999996e-06 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952911+0000", 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027069099999999999 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952989+0000", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8485000000000005e-05 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.953127+0000", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746799999999999 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957228+0000", 2024-09-13T13:51:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041009810000000001 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957981+0000", 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075359999999999999 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958037+0000", 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5349999999999997e-05 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958078+0000", 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1553e-05 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.732842771000001, 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.082 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:51:06.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:51:06.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:06.087 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.732756454, 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:51:06.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.732696084000001, 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.093 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.098843939, 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:51:06.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:51:06.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2633 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 1660464~1506637] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.15377339200000001, 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083516809999999997, 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2633 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.105 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:06.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821147+0000", 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821153+0000", 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821158+0000", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5820000000000001e-06 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821293+0000", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013434499999999999 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821298+0000", 2024-09-13T13:51:06.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.828410+0000", 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071116349999999998 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.828495+0000", 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5327000000000005e-05 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829496+0000", 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001000754 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2634 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 0~2456909] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.15374592300000001, 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087606320000000008, 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.110 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2634 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821172+0000", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821175+0000", 2024-09-13T13:51:06.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5719999999999999e-06 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821180+0000", 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821185+0000", 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6590000000000004e-06 2024-09-13T13:51:06.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821322+0000", 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013670500000000001 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821328+0000", 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8170000000000004e-06 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829522+0000", 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081938299999999992 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829600+0000", 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7750000000000006e-05 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829932+0000", 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033246000000000003 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.116 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.116 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: finishing write tid 1 to smithi15938319-4 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: finishing write tid 2 to smithi15938319-4 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: finishing write tid 3 to smithi15938319-4 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: finishing write tid 4 to smithi15938319-4 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: finishing write tid 5 to smithi15938319-4 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: oid 4 updating version 0 to 150 2024-09-13T13:51:06.117 INFO:tasks.rados.rados.0.smithi159.stdout:1260: oid 4 updating version 150 to 151 2024-09-13T13:51:06.118 INFO:tasks.rados.rados.0.smithi159.stdout:1260: oid 4 version 151 is already newer than 148 2024-09-13T13:51:06.118 INFO:tasks.rados.rados.0.smithi159.stdout:1260: oid 4 version 151 is already newer than 149 2024-09-13T13:51:06.118 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 151 (ObjNum 214 snap 23 seq_num 214) dirty exists 2024-09-13T13:51:06.118 INFO:tasks.rados.rados.0.smithi159.stdout:1260: left oid 4 (ObjNum 214 snap 23 seq_num 214) 2024-09-13T13:51:06.119 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:06.119 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:06.117+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:06.218 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:06.217+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:06.263 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:06.263 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:06.320 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:06.319+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:06.374 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.374 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:06.374 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:06.375 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:06.375 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:06.421 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:06.419+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:06.486 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 52.715427861000002, 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.487 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:51:06.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 52.706397428999999, 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:51:06.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 48.054391553000002, 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.498 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:51:06.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:51:06.502 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 48.051357566999997, 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:51:06.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:51:06.506 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.175525448999998, 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:51:06.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:51:06.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.169618617000001, 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:51:06.516 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.162183182, 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:51:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:51:06.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.242889439999999, 2024-09-13T13:51:06.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:51:06.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:06.527 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.242820774999998, 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:06.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:51:06.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.242770661000002, 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:51:06.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:51:06.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:51:06.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.221813300000001, 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:51:06.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:51:06.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.216217097000001, 2024-09-13T13:51:06.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:06.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:51:06.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.214673186999999, 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:51:06.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:51:06.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.214608138999999, 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:06.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:51:06.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.60766654, 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.565 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:51:06.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:51:06.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.751990836999999, 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:51:06.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:06.575 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.680095149, 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:51:06.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:51:06.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.680057628, 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:51:06.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:51:06.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.567138872999999, 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.587 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:51:06.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:51:06.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.353072465, 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:06.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:06.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:51:06.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.600 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.602 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:06.602 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:06.601+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:06.703 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:06.702+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:06.704 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.704 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:06.705 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:06.803 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:06.803+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:06.816 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.816 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:06.816 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:06.816 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:06.816 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.822 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:06.822+0000 7f1a526ee640 -1 osd.13 89 handle_osd_map injecting map crc failure 2024-09-13T13:51:06.825 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1256: done (5 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1257: done (4 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1258: done (3 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1259: done (2 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1260: done (1 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1261: done (0 left) 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1262: write (excl) oid 30 current snap is 24 2024-09-13T13:51:06.826 INFO:tasks.rados.rados.0.smithi159.stdout:1262: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-13T13:51:06.831 INFO:tasks.rados.rados.0.smithi159.stdout:1262: writing smithi15938319-30 from 413505 to 1049924 tid 1 2024-09-13T13:51:06.838 INFO:tasks.rados.rados.0.smithi159.stdout:1262: writing smithi15938319-30 from 1664805 to 2353869 tid 2 2024-09-13T13:51:06.844 INFO:tasks.rados.rados.0.smithi159.stdout:1262: writing smithi15938319-30 from 3038061 to 3530132 tid 3 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1263: read oid 15 snap -1 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1263: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1264: read oid 32 snap -1 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1264: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1265: delete oid 28 current snap is 24 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1262: finishing write tid 1 to smithi15938319-30 2024-09-13T13:51:06.845 INFO:tasks.rados.rados.0.smithi159.stdout:1262: finishing write tid 2 to smithi15938319-30 2024-09-13T13:51:06.845 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:06.844+0000 7f9b0e557640 -1 osd.12 89 handle_osd_map injecting map crc failure 2024-09-13T13:51:06.849 INFO:tasks.rados.rados.0.smithi159.stdout:1265: done (3 left) 2024-09-13T13:51:06.849 INFO:tasks.rados.rados.0.smithi159.stdout:1266: delete oid 36 current snap is 24 2024-09-13T13:51:06.849 INFO:tasks.rados.rados.0.smithi159.stdout:1262: finishing write tid 3 to smithi15938319-30 2024-09-13T13:51:06.849 INFO:tasks.rados.rados.0.smithi159.stdout:1262: finishing write tid 4 to smithi15938319-30 2024-09-13T13:51:06.849 INFO:tasks.rados.rados.0.smithi159.stdout:1262: finishing write tid 5 to smithi15938319-30 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1262: oid 30 updating version 0 to 154 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1262: oid 30 version 154 is already newer than 153 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1262: oid 30 version 154 is already newer than 152 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1262: oid 30 updating version 154 to 155 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 155 (ObjNum 215 snap 24 seq_num 215) dirty exists 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1262: left oid 30 (ObjNum 215 snap 24 seq_num 215) 2024-09-13T13:51:06.850 INFO:tasks.rados.rados.0.smithi159.stdout:1264: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:51:06.905 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:06.904+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:06.929 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:06.929 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 75.695007031000003, 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:06.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:51:06.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 74.686972885000003, 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:06.935 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:51:06.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:06.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.939 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.291274726000001, 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:51:06.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:06.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.943 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.944 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.944 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:06.944 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:06.944 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:06.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:07.004 INFO:tasks.rados.rados.0.smithi159.stdout:1263: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:07.005 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:07.006 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:07.004+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:07.049 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:07.049 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:07.049 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:07.049 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:07.057 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:07.094 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:07.107 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:07.106+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1262: done (3 left) 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1263: done (2 left) 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1264: done (1 left) 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1266: done (0 left) 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1267: write (excl) oid 50 current snap is 24 2024-09-13T13:51:07.147 INFO:tasks.rados.rados.0.smithi159.stdout:1267: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-13T13:51:07.151 INFO:tasks.rados.rados.0.smithi159.stdout:1267: writing smithi15938319-50 from 463926 to 926092 tid 1 2024-09-13T13:51:07.158 INFO:tasks.rados.rados.0.smithi159.stdout:1267: writing smithi15938319-50 from 1723898 to 2160289 tid 2 2024-09-13T13:51:07.159 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:07.159 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:07.165 INFO:tasks.rados.rados.0.smithi159.stdout:1267: writing smithi15938319-50 from 2603977 to 3329864 tid 3 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1268: read oid 17 snap 20 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1268: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1269: rollback oid 3 current snap is 24 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 3 to 22 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1267: finishing write tid 1 to smithi15938319-50 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1267: finishing write tid 2 to smithi15938319-50 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1270: read oid 37 snap 17 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1270: expect (ObjNum 154 snap 13 seq_num 154) 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1271: read oid 12 snap -1 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1271: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:07.166 INFO:tasks.rados.rados.0.smithi159.stdout:1272: write oid 48 current snap is 24 2024-09-13T13:51:07.167 INFO:tasks.rados.rados.0.smithi159.stdout:1272: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-13T13:51:07.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:07.170 INFO:tasks.rados.rados.0.smithi159.stdout:1272: writing smithi15938319-48 from 480733 to 1018148 tid 1 2024-09-13T13:51:07.179 INFO:tasks.rados.rados.0.smithi159.stdout:1272: writing smithi15938319-48 from 1782380 to 2478782 tid 2 2024-09-13T13:51:07.185 INFO:tasks.rados.rados.0.smithi159.stdout:1272: writing smithi15938319-48 from 3031400 to 3568559 tid 3 2024-09-13T13:51:07.185 INFO:tasks.rados.rados.0.smithi159.stdout:1273: write (excl) oid 32 current snap is 24 2024-09-13T13:51:07.185 INFO:tasks.rados.rados.0.smithi159.stdout:1273: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-13T13:51:07.191 INFO:tasks.rados.rados.0.smithi159.stdout:1273: writing smithi15938319-32 from 497540 to 1110204 tid 1 2024-09-13T13:51:07.197 INFO:tasks.rados.rados.0.smithi159.stdout:1273: writing smithi15938319-32 from 1840862 to 2280922 tid 2 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1273: writing smithi15938319-32 from 3058823 to 3523607 tid 3 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1274: read oid 47 snap -1 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1274: expect (ObjNum 205 snap 21 seq_num 205) 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1275: rollback oid 25 current snap is 24 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 25 to 21 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1269: finishing rollback tid 0 to smithi15938319-3 2024-09-13T13:51:07.205 INFO:tasks.rados.rados.0.smithi159.stdout:1268: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:51:07.207 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:07.206+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:07.273 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:07.273 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:07.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:07.273 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:07.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.504253183000003, 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:51:07.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:51:07.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.503275985000002, 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:51:07.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:51:07.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:51:07.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.286 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.501521320000002, 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:51:07.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:51:07.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.293 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.501129106999997, 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:51:07.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.299 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.501091780000003, 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:07.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:51:07.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:51:07.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:07.306 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.496293426999998, 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:51:07.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:51:07.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.480658202000001, 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.313 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:51:07.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.480551851999998, 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.319 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:51:07.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:51:07.325 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.472618081999997, 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.326 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:51:07.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:51:07.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.454355337999999, 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.333 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:07.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:51:07.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.454229523999999, 2024-09-13T13:51:07.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:07.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:51:07.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.454102734000003, 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:51:07.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:51:07.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:51:07.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.352 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.372904310999999, 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:51:07.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:51:07.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.358 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.37280286, 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:07.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:51:07.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:51:07.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 42.811450094000001, 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.365 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:51:07.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:51:07.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:07.371 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 42.805598551000003, 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:07.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.375 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.528788915, 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:07.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 33.543721099999999, 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:51:07.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:51:07.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:07.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.387 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.537699824999997, 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:07.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:51:07.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.393 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2394 3.1 3:93ff5947:::3634ac6462bf9cb96fe06c81293bb3c6c0e95892f99c8f55c0a14944e78c533e:head [write 0~1120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.492311967999999, 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012999503000000001, 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:07.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2394 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:07.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778747+0000", 2024-09-13T13:51:07.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.886e-06 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780479+0000", 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017320300000000001 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780496+0000", 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7070000000000001e-05 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780522+0000", 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6588999999999999e-05 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780613+0000", 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0382000000000003e-05 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780619+0000", 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.353e-06 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786863+0000", 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062442749999999997 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786980+0000", 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116236 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:07.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.787126+0000", 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014658299999999999 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791421+0000", 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042944840000000003 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791672+0000", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025098399999999999 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791713+0000", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1306000000000001e-05 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791744+0000", 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1325000000000002e-05 2024-09-13T13:51:07.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:07.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:07.403 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:07.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: finishing write tid 3 to smithi15938319-50 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: finishing write tid 4 to smithi15938319-50 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1269: finishing rollback tid 1 to smithi15938319-3 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: finishing write tid 5 to smithi15938319-50 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: oid 50 updating version 0 to 164 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: oid 50 updating version 164 to 166 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: oid 50 version 166 is already newer than 163 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: oid 50 version 166 is already newer than 165 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 166 (ObjNum 218 snap 24 seq_num 218) dirty exists 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: left oid 50 (ObjNum 218 snap 24 seq_num 218) 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1267: done (8 left) 2024-09-13T13:51:07.406 INFO:tasks.rados.rados.0.smithi159.stdout:1268: done (7 left) 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1276: read oid 36 snap -1 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1276: expect deleted 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1277: read oid 49 snap -1 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1277: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1278: write oid 34 current snap is 24 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1269: finishing rollback tid 2 to smithi15938319-3 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 159 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:1270: expect (ObjNum 154 snap 13 seq_num 154) 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:07.407 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1402259 2024-09-13T13:51:07.407 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:07.317+0000 7f002dbdb640 -1 osd.15 89 handle_osd_map injecting map crc failure 2024-09-13T13:51:07.408 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1842994 2024-09-13T13:51:07.408 INFO:tasks.rados.rados.0.smithi159.stdout:1272: finishing write tid 1 to smithi15938319-48 2024-09-13T13:51:07.408 INFO:tasks.rados.rados.0.smithi159.stdout:1271: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:07.408 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1953134 2024-09-13T13:51:07.408 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:07.409 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:07.403+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:07.491 INFO:tasks.rados.rados.0.smithi159.stdout:1272: finishing write tid 2 to smithi15938319-48 2024-09-13T13:51:07.491 INFO:tasks.rados.rados.0.smithi159.stdout:1272: finishing write tid 3 to smithi15938319-48 2024-09-13T13:51:07.491 INFO:tasks.rados.rados.0.smithi159.stdout:1272: finishing write tid 4 to smithi15938319-48 2024-09-13T13:51:07.491 INFO:tasks.rados.rados.0.smithi159.stdout:1272: finishing write tid 5 to smithi15938319-48 2024-09-13T13:51:07.491 INFO:tasks.rados.rados.0.smithi159.stdout:1272: oid 48 updating version 0 to 183 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1272: oid 48 version 183 is already newer than 182 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1272: oid 48 updating version 183 to 184 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1272: oid 48 updating version 184 to 185 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 185 (ObjNum 219 snap 24 seq_num 219) dirty exists 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1272: left oid 48 (ObjNum 219 snap 24 seq_num 219) 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1273: finishing write tid 1 to smithi15938319-32 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1273: finishing write tid 2 to smithi15938319-32 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1275: finishing rollback tid 0 to smithi15938319-25 2024-09-13T13:51:07.492 INFO:tasks.rados.rados.0.smithi159.stdout:1278: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-13T13:51:07.496 INFO:tasks.rados.rados.0.smithi159.stdout:1278: writing smithi15938319-34 from 514347 to 918613 tid 1 2024-09-13T13:51:07.503 INFO:tasks.rados.rados.0.smithi159.stdout:1278: writing smithi15938319-34 from 1499344 to 2199415 tid 2 2024-09-13T13:51:07.505 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:07.504+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:07.509 INFO:tasks.rados.rados.0.smithi159.stdout:1278: writing smithi15938319-34 from 2686246 to 3362302 tid 3 2024-09-13T13:51:07.509 INFO:tasks.rados.rados.0.smithi159.stdout:1269: done (9 left) 2024-09-13T13:51:07.509 INFO:tasks.rados.rados.0.smithi159.stdout:1270: done (8 left) 2024-09-13T13:51:07.510 INFO:tasks.rados.rados.0.smithi159.stdout:1271: done (7 left) 2024-09-13T13:51:07.510 INFO:tasks.rados.rados.0.smithi159.stdout:1272: done (6 left) 2024-09-13T13:51:07.510 INFO:tasks.rados.rados.0.smithi159.stdout:1279: write (excl) oid 41 current snap is 24 2024-09-13T13:51:07.510 INFO:tasks.rados.rados.0.smithi159.stdout:1279: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-13T13:51:07.514 INFO:tasks.rados.rados.0.smithi159.stdout:1279: writing smithi15938319-41 from 531154 to 1010669 tid 1 2024-09-13T13:51:07.519 INFO:tasks.rados.rados.0.smithi159.stdout:1279: writing smithi15938319-41 from 1557826 to 2001555 tid 2 2024-09-13T13:51:07.525 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:07.525 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:07.525 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:07.526 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:07.526 INFO:tasks.rados.rados.0.smithi159.stdout:1279: writing smithi15938319-41 from 2713669 to 3200997 tid 3 2024-09-13T13:51:07.529 INFO:tasks.rados.rados.0.smithi159.stdout:1279: writing smithi15938319-41 from 3679295 to 3731154 tid 4 2024-09-13T13:51:07.530 INFO:tasks.rados.rados.0.smithi159.stdout:1280: read oid 20 snap -1 2024-09-13T13:51:07.530 INFO:tasks.rados.rados.0.smithi159.stdout:1280: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:51:07.530 INFO:tasks.rados.rados.0.smithi159.stdout:1281: set_chunk oid 7 target oid 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1281: set_chunk oid 7 offset: 2525184 length: 292352 target oid 94b3820ca87c3f8b7b585b93123ae11fb24caf4d080d516f049d929825542906 offset: 0 snap 22 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1282: tier_evict oid 29 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1282: tier_evict oid 29 snap -1 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1275: finishing rollback tid 1 to smithi15938319-25 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1275: finishing rollback tid 2 to smithi15938319-25 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 100 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:51:07.534 INFO:tasks.rados.rados.0.smithi159.stdout:1273: finishing write tid 3 to smithi15938319-32 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1275: done (9 left) 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1283: read oid 17 snap -1 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1283: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1284: read oid 18 snap -1 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1284: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:07.535 INFO:tasks.rados.rados.0.smithi159.stdout:1285: set_chunk oid 15 target oid 2024-09-13T13:51:07.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:07.554 INFO:tasks.rados.rados.0.smithi159.stdout:1285: set_chunk oid 15 offset: 44544 length: 1945088 target oid a517de21ad54460ea17f68e67289b454638a48bcfc3f4811dbab2341b0c1950e offset: 0 snap -1 2024-09-13T13:51:07.555 INFO:tasks.rados.rados.0.smithi159.stdout:1286: write (excl) oid 27 current snap is 24 2024-09-13T13:51:07.555 INFO:tasks.rados.rados.0.smithi159.stdout:1273: finishing write tid 4 to smithi15938319-32 2024-09-13T13:51:07.555 INFO:tasks.rados.rados.0.smithi159.stdout:1273: finishing write tid 5 to smithi15938319-32 2024-09-13T13:51:07.555 INFO:tasks.rados.rados.0.smithi159.stdout:1273: oid 32 updating version 0 to 90 2024-09-13T13:51:07.555 INFO:tasks.rados.rados.0.smithi159.stdout:1273: oid 32 updating version 90 to 92 2024-09-13T13:51:07.556 INFO:tasks.rados.rados.0.smithi159.stdout:1273: oid 32 updating version 92 to 93 2024-09-13T13:51:07.556 INFO:tasks.rados.rados.0.smithi159.stdout:1273: oid 32 version 93 is already newer than 91 2024-09-13T13:51:07.556 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 93 (ObjNum 220 snap 24 seq_num 220) dirty exists 2024-09-13T13:51:07.556 INFO:tasks.rados.rados.0.smithi159.stdout:1273: left oid 32 (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:07.556 INFO:tasks.rados.rados.0.smithi159.stdout:1274: expect (ObjNum 205 snap 21 seq_num 205) 2024-09-13T13:51:07.559 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1070717 2024-09-13T13:51:07.606 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:07.605+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:07.631 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2319528 2024-09-13T13:51:07.639 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:07.639 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:07.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:07.666 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 48000 2024-09-13T13:51:07.707 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:07.706+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:07.732 INFO:tasks.rados.rados.0.smithi159.stdout:1277: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:07.752 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:07.752 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:07.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.852669693000003, 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.753 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:51:07.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:51:07.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:265 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.839390385999998, 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082114400000000004, 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 265 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911445+0000", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911447+0000", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911457+0000", 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0433e-05 2024-09-13T13:51:07.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911476+0000", 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8612000000000001e-05 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911486+0000", 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0105999999999999e-05 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911546+0000", 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9877000000000003e-05 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.911627+0000", 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1546999999999997e-05 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.917585+0000", 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059573229999999996 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.918841+0000", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001255969 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919557+0000", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000716079 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919613+0000", 2024-09-13T13:51:07.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5881000000000003e-05 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.919656+0000", 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3643999999999998e-05 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.837042078000003, 2024-09-13T13:51:07.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:07.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:51:07.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.771 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.828330403000002, 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:51:07.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:51:07.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.980784440000001, 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:51:07.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:51:07.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:366 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.6759147, 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079177420000000002, 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 366 2024-09-13T13:51:07.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074920+0000", 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074921+0000", 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074925+0000", 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8010000000000001e-06 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074933+0000", 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1880000000000002e-06 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.074939+0000", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075137+0000", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197927 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.075206+0000", 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8806999999999994e-05 2024-09-13T13:51:07.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.078002+0000", 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002796078 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.081901+0000", 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003899713 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082743+0000", 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084112100000000001 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082793+0000", 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0359999999999999e-05 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:14.082838+0000", 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5321000000000002e-05 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.789 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1038 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-promote] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.970811406999999, 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078733310000000008, 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.790 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1038 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780024+0000", 2024-09-13T13:51:07.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780025+0000", 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.238e-06 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780029+0000", 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1470000000000003e-06 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780040+0000", 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0597e-05 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780049+0000", 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3260000000000002e-06 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780132+0000", 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3185999999999999e-05 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.780189+0000", 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7234000000000001e-05 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:23.787897+0000", 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077076030000000004 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 39.004044253000004, 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:51:07.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:51:07.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing(4.f epoch 53/34 rep_tid 118 entries 53'57 (0'0) error 4:f883ee18:::smithi15938319-40:head by client.4571.0:1290 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 51'54)", 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 39.001208247000001, 2024-09-13T13:51:07.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084851660000000006, 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749627+0000", 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749628+0000", 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0400000000000005e-07 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749637+0000", 2024-09-13T13:51:07.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6080000000000002e-06 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749672+0000", 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4409999999999998e-05 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749685+0000", 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2754e-05 2024-09-13T13:51:07.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.754728+0000", 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050430300000000004 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758112+0000", 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033844600000000002 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 37.789373974, 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.804 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:51:07.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.809 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 34.787572347999998, 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:51:07.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:51:07.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1802 4.1 4:8bb695d2:::smithi15938319-48:c [read 3113149~634779,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.508701598999998, 2024-09-13T13:51:07.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083353070000000001, 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1802 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:07.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242134+0000", 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242135+0000", 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.48e-06 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242142+0000", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6000000000000003e-06 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242164+0000", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2356000000000001e-05 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242173+0000", 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3279999999999995e-06 2024-09-13T13:51:07.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242337+0000", 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016404000000000001 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242349+0000", 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1440999999999999e-05 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245783+0000", 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034342190000000001 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245937+0000", 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250469+0000", 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045318429999999998 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.819 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.508648781000002, 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.820 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:51:07.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:51:07.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.508625422000001, 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.825 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:51:07.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.830 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.14.0:137 4.b e64/34 4:def5de2b:::smithi15938319-3:head v 64'70, mlcod=61'68)", 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.490548169, 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0080519279999999999, 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260287+0000", 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8000000000000002e-07 2024-09-13T13:51:07.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260942+0000", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065496 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260991+0000", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8848999999999999e-05 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261001+0000", 2024-09-13T13:51:07.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8749999999999995e-06 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261068+0000", 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7076999999999995e-05 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.261091+0000", 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2648999999999999e-05 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.268297+0000", 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072060279999999997 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.268339+0000", 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2209999999999997e-05 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.835 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.010180585000001, 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:51:07.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:51:07.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:51:07.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:07.842 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.9245592220000001, 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:07.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:51:07.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:2640 4.b e89/34 4:d4fa90c1:::smithi15938319-4:head v 89'148, mlcod=87'144)", 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:05.839596+0000", 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.911239006, 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082401609999999993, 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.847 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.839596+0000", 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.839596+0000", 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.839596+0000", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9799999999999999e-07 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.840083+0000", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00048620100000000002 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.840144+0000", 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1229999999999995e-05 2024-09-13T13:51:07.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.840154+0000", 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7200000000000001e-06 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.840218+0000", 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4465e-05 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.840245+0000", 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6653e-05 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.847795+0000", 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075497660000000003 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:05.847836+0000", 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1828e-05 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.851 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2676 4.1 4:8bb695d2:::smithi15938319-48:head [write 480733~537415] snapc 19=[19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:07.170924+0000", 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.57991127600000003, 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084919769999999995, 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2676 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.170924+0000", 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.170924+0000", 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.170929+0000", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0429999999999998e-06 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175527+0000", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045981729999999997 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175539+0000", 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2136999999999999e-05 2024-09-13T13:51:07.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175563+0000", 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4128999999999999e-05 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175622+0000", 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8628000000000002e-05 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175689+0000", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7071999999999998e-05 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.175817+0000", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012839700000000001 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.178260+0000", 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024429170000000002 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179339+0000", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010785 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179380+0000", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0902999999999998e-05 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179416+0000", 2024-09-13T13:51:07.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6078000000000003e-05 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2677 4.1 4:8bb695d2:::smithi15938319-48:head [write 1782380~696402] snapc 19=[19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:07.179270+0000", 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.57156529199999995, 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0086581469999999997, 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:07.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2677 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179270+0000", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179270+0000", 2024-09-13T13:51:07.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.179271+0000", 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.246e-06 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180074+0000", 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000802552 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180087+0000", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2979999999999999e-05 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180094+0000", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8940000000000004e-06 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180137+0000", 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3736000000000003e-05 2024-09-13T13:51:07.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180194+0000", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6691999999999999e-05 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.180280+0000", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6067000000000001e-05 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.184377+0000", 2024-09-13T13:51:07.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040973420000000003 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.187829+0000", 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034517380000000002 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.187897+0000", 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7402000000000003e-05 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:07.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:07.187928+0000", 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1498000000000002e-05 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:07.864 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:07.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:07.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:51:07.867 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:07.868 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: finishing write tid 1 to smithi15938319-34 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: finishing write tid 2 to smithi15938319-34 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: finishing write tid 3 to smithi15938319-34 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: finishing write tid 4 to smithi15938319-34 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: finishing write tid 5 to smithi15938319-34 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: oid 34 updating version 0 to 113 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: oid 34 updating version 113 to 114 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: oid 34 updating version 114 to 115 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: oid 34 updating version 115 to 116 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 116 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1278: left oid 34 (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 1 to smithi15938319-41 2024-09-13T13:51:07.869 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 2 to smithi15938319-41 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1286: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1286: writing smithi15938319-27 from 547961 to 1102725 tid 1 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1286: writing smithi15938319-27 from 1616308 to 2320048 tid 2 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1286: writing smithi15938319-27 from 2741092 to 3439692 tid 3 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1273: done (12 left) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1274: done (11 left) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1276: done (10 left) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1277: done (9 left) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1278: done (8 left) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1287: read oid 3 snap -1 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1287: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1288: read oid 2 snap 20 2024-09-13T13:51:07.870 INFO:tasks.rados.rados.0.smithi159.stdout:1288: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1289: read oid 8 snap 20 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1289: expect deleted 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1290: read oid 30 snap 22 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1290: expect deleted 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1291: write (excl) oid 35 current snap is 24 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 3 to smithi15938319-41 2024-09-13T13:51:07.871 INFO:tasks.rados.rados.0.smithi159.stdout:1291: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-13T13:51:07.871 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:07.865+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:07.872 INFO:tasks.rados.rados.0.smithi159.stdout:1291: writing smithi15938319-35 from 564768 to 1194781 tid 1 2024-09-13T13:51:07.873 INFO:tasks.rados.rados.0.smithi159.stdout:1291: writing smithi15938319-35 from 1674790 to 2122188 tid 2 2024-09-13T13:51:07.880 INFO:tasks.rados.rados.0.smithi159.stdout:1291: writing smithi15938319-35 from 2768515 to 3394740 tid 3 2024-09-13T13:51:07.880 INFO:tasks.rados.rados.0.smithi159.stdout:1292: read oid 42 snap -1 2024-09-13T13:51:07.880 INFO:tasks.rados.rados.0.smithi159.stdout:1292: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:07.880 INFO:tasks.rados.rados.0.smithi159.stdout:1293: write (excl) oid 47 current snap is 24 2024-09-13T13:51:07.881 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 4 to smithi15938319-41 2024-09-13T13:51:07.881 INFO:tasks.rados.rados.0.smithi159.stdout:1293: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-13T13:51:07.887 INFO:tasks.rados.rados.0.smithi159.stdout:1293: writing smithi15938319-47 from 581575 to 1286837 tid 1 2024-09-13T13:51:07.893 INFO:tasks.rados.rados.0.smithi159.stdout:1293: writing smithi15938319-47 from 1733272 to 2440681 tid 2 2024-09-13T13:51:07.900 INFO:tasks.rados.rados.0.smithi159.stdout:1293: writing smithi15938319-47 from 3195938 to 3633435 tid 3 2024-09-13T13:51:07.900 INFO:tasks.rados.rados.0.smithi159.stdout:1294: read oid 46 snap -1 2024-09-13T13:51:07.900 INFO:tasks.rados.rados.0.smithi159.stdout:1294: expect deleted 2024-09-13T13:51:07.900 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:07.900 INFO:tasks.rados.rados.0.smithi159.stdout:1280: expect (ObjNum 183 snap 17 seq_num 183) 2024-09-13T13:51:07.966 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:07.966+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:07.976 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:07.977 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:07.977 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:07.977 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:07.986 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:08.022 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:08.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:08.045 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:08.045 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:08.066 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:08.066 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:08.067 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:08.066+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:08.070 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:08.069+0000 7f9b0c553640 -1 osd.12 90 handle_osd_map injecting map crc failure 2024-09-13T13:51:08.087 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:08.098 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 5 to smithi15938319-41 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: finishing write tid 6 to smithi15938319-41 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: oid 41 updating version 0 to 153 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: oid 41 updating version 153 to 155 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: oid 41 version 155 is already newer than 154 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: oid 41 updating version 155 to 156 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: oid 41 updating version 156 to 157 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 157 (ObjNum 222 snap 24 seq_num 222) dirty exists 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1279: left oid 41 (ObjNum 222 snap 24 seq_num 222) 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1282: got (22) Invalid argument 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1281: finishing set_chunk to oid 7 2024-09-13T13:51:08.099 INFO:tasks.rados.rados.0.smithi159.stdout:1283: expect (ObjNum 127 snap 9 seq_num 127) 2024-09-13T13:51:08.108 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:08.108 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:08.120 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:08.128 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:08.128 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:08.129 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:08.129 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:08.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:08.150 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:08.170 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:08.171 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:08.178 INFO:tasks.rados.rados.0.smithi159.stdout:1284: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:08.182 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:08.180+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:08.192 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:08.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1613797 2024-09-13T13:51:08.213 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:08.235 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:08.235 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:08.240 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.241 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:08.251 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:08.256 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:08.276 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:08.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:08.275+0000 7fd848994640 0 -- 172.21.15.27:0/3835734927 >> v2:172.21.15.100:6800/450451503 conn(0x7fd84421cbe0 msgr2=0x7fd84421efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:08.291 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:08.289+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:08.333 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:08.369 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:08.368+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:08.393 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.393 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:08.416 INFO:tasks.rados.rados.0.smithi159.stdout:1285: finishing set_chunk to oid 15 2024-09-13T13:51:08.416 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 44544 length: 1945088 2024-09-13T13:51:08.416 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:51:08.416 INFO:tasks.rados.rados.0.smithi159.stdout:1279: done (15 left) 2024-09-13T13:51:08.416 INFO:tasks.rados.rados.0.smithi159.stdout:1280: done (14 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1281: done (13 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1282: done (12 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1283: done (11 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1284: done (10 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1285: done (9 left) 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1295: set_chunk oid 36 target oid 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1295: set_chunk oid 36 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:08.417 INFO:tasks.rados.rados.0.smithi159.stdout:1296: set_chunk oid 32 target oid 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1296: set_chunk oid 32 offset: 814592 length: 1472512 target oid 9ac395ff1853ef52fcaf13deef110f9584a81f40a5a6d3c5d8cfe1776aa07f4b offset: 0 snap -1 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1297: delete oid 50 current snap is 24 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: finishing write tid 1 to smithi15938319-27 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: finishing write tid 2 to smithi15938319-27 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: finishing write tid 3 to smithi15938319-27 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: finishing write tid 4 to smithi15938319-27 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: finishing write tid 5 to smithi15938319-27 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: oid 27 updating version 0 to 163 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: oid 27 updating version 163 to 164 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: oid 27 version 164 is already newer than 162 2024-09-13T13:51:08.426 INFO:tasks.rados.rados.0.smithi159.stdout:1286: oid 27 version 164 is already newer than 161 2024-09-13T13:51:08.427 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 164 (ObjNum 223 snap 24 seq_num 223) dirty exists 2024-09-13T13:51:08.427 INFO:tasks.rados.rados.0.smithi159.stdout:1286: left oid 27 (ObjNum 223 snap 24 seq_num 223) 2024-09-13T13:51:08.427 INFO:tasks.rados.rados.0.smithi159.stdout:1287: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-13T13:51:08.435 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1745940 2024-09-13T13:51:08.442 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 899541 2024-09-13T13:51:08.454 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:08.469 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:08.469+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:08.477 INFO:tasks.rados.rados.0.smithi159.stdout:1291: finishing write tid 1 to smithi15938319-35 2024-09-13T13:51:08.477 INFO:tasks.rados.rados.0.smithi159.stdout:1288: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:08.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:08.499 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:08.505 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.505 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:08.521 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:08.521 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:08.542 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:08.542 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:08.570 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:08.568+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:08.616 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.616 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.624 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2527446 2024-09-13T13:51:08.625 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: finishing write tid 2 to smithi15938319-35 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: finishing write tid 3 to smithi15938319-35 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: finishing write tid 4 to smithi15938319-35 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: finishing write tid 5 to smithi15938319-35 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: oid 35 updating version 0 to 188 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: oid 35 updating version 188 to 189 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: oid 35 updating version 189 to 190 2024-09-13T13:51:08.637 INFO:tasks.rados.rados.0.smithi159.stdout:1291: oid 35 version 190 is already newer than 187 2024-09-13T13:51:08.638 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 190 (ObjNum 224 snap 24 seq_num 224) dirty exists 2024-09-13T13:51:08.638 INFO:tasks.rados.rados.0.smithi159.stdout:1291: left oid 35 (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:08.638 INFO:tasks.rados.rados.0.smithi159.stdout:1292: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:08.697 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 894180 2024-09-13T13:51:08.709 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:08.707+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:51:08.770 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.770 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.772 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:08.769+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:08.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:08.808 INFO:tasks.rados.rados.0.smithi159.stdout:1293: finishing write tid 1 to smithi15938319-47 2024-09-13T13:51:08.808 INFO:tasks.rados.rados.0.smithi159.stdout:1293: finishing write tid 2 to smithi15938319-47 2024-09-13T13:51:08.808 INFO:tasks.rados.rados.0.smithi159.stdout:1293: finishing write tid 3 to smithi15938319-47 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: finishing write tid 4 to smithi15938319-47 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: finishing write tid 5 to smithi15938319-47 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: oid 47 updating version 0 to 119 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: oid 47 version 119 is already newer than 118 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: oid 47 updating version 119 to 121 2024-09-13T13:51:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:1293: oid 47 version 121 is already newer than 120 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 121 (ObjNum 225 snap 24 seq_num 225) dirty exists 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1293: left oid 47 (ObjNum 225 snap 24 seq_num 225) 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1286: done (11 left) 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1287: done (10 left) 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1288: done (9 left) 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1289: done (8 left) 2024-09-13T13:51:08.810 INFO:tasks.rados.rados.0.smithi159.stdout:1290: done (7 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1291: done (6 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1292: done (5 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1293: done (4 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1294: done (3 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1297: done (2 left) 2024-09-13T13:51:08.811 INFO:tasks.rados.rados.0.smithi159.stdout:1298: snap_create 2024-09-13T13:51:08.812 INFO:tasks.rados.rados.0.smithi159.stdout:1295: finishing set_chunk to oid 36 2024-09-13T13:51:08.812 INFO:tasks.rados.rados.0.smithi159.stdout:1295: got expected ERANGE (src dne) 2024-09-13T13:51:08.812 INFO:tasks.rados.rados.0.smithi159.stdout:1296: finishing set_chunk to oid 32 2024-09-13T13:51:08.812 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 93 (ObjNum 220 snap 24 seq_num 220) dirty exists 2024-09-13T13:51:08.873 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:08.872+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:08.924 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:08.925 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:08.933 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:08.974 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:08.973+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:09.037 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.037 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.046 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:09.075 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:09.074+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:09.149 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.149 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:09.175 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:09.175+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:09.261 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.261 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:09.314 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:09.313+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:09.373 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.373 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.377 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:09.375+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:09.382 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:09.478 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:09.478+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:09.526 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.526 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.535 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:09.580 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:09.579+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:09.637 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.637 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.646 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:09.681 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:09.680+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:09.732 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:09.730+0000 7f2284f10640 -1 osd.10 92 handle_osd_map injecting map crc failure 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1295: done (2 left) 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1296: done (1 left) 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1298: done (0 left) 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1299: read oid 23 snap 22 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1299: expect (ObjNum 163 snap 13 seq_num 163) 2024-09-13T13:51:09.735 INFO:tasks.rados.rados.0.smithi159.stdout:1300: tier_promote oid 36 2024-09-13T13:51:09.736 INFO:tasks.rados.rados.0.smithi159.stdout:1301: write (excl) oid 10 current snap is 25 2024-09-13T13:51:09.736 INFO:tasks.rados.rados.0.smithi159.stdout:1301: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-13T13:51:09.741 INFO:tasks.rados.rados.0.smithi159.stdout:1301: writing smithi15938319-10 from 615189 to 1070949 tid 1 2024-09-13T13:51:09.748 INFO:tasks.rados.rados.0.smithi159.stdout:1301: writing smithi15938319-10 from 1733883 to 2444961 tid 2 2024-09-13T13:51:09.750 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.750 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.755 INFO:tasks.rados.rados.0.smithi159.stdout:1301: writing smithi15938319-10 from 3134431 to 3594472 tid 3 2024-09-13T13:51:09.755 INFO:tasks.rados.rados.0.smithi159.stdout:1302: write (excl) oid 19 current snap is 25 2024-09-13T13:51:09.755 INFO:tasks.rados.rados.0.smithi159.stdout:1302: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-13T13:51:09.759 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:09.761 INFO:tasks.rados.rados.0.smithi159.stdout:1302: writing smithi15938319-19 from 631996 to 1163005 tid 1 2024-09-13T13:51:09.767 INFO:tasks.rados.rados.0.smithi159.stdout:1302: writing smithi15938319-19 from 1792365 to 2247101 tid 2 2024-09-13T13:51:09.774 INFO:tasks.rados.rados.0.smithi159.stdout:1302: writing smithi15938319-19 from 2761854 to 3433167 tid 3 2024-09-13T13:51:09.774 INFO:tasks.rados.rados.0.smithi159.stdout:1303: write (excl) oid 3 current snap is 25 2024-09-13T13:51:09.774 INFO:tasks.rados.rados.0.smithi159.stdout:1303: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-13T13:51:09.781 INFO:tasks.rados.rados.0.smithi159.stdout:1303: writing smithi15938319-3 from 648803 to 1371414 tid 1 2024-09-13T13:51:09.781 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:09.779+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:09.788 INFO:tasks.rados.rados.0.smithi159.stdout:1303: writing smithi15938319-3 from 1967200 to 2681947 tid 2 2024-09-13T13:51:09.794 INFO:tasks.rados.rados.0.smithi159.stdout:1303: writing smithi15938319-3 from 3421983 to 3848803 tid 3 2024-09-13T13:51:09.795 INFO:tasks.rados.rados.0.smithi159.stdout:1304: write oid 25 current snap is 25 2024-09-13T13:51:09.795 INFO:tasks.rados.rados.0.smithi159.stdout:1299: expect (ObjNum 163 snap 13 seq_num 163) 2024-09-13T13:51:09.808 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2494417 2024-09-13T13:51:09.864 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.864 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:09.874 INFO:tasks.rados.rados.0.smithi159.stdout:1300: got (0) Success 2024-09-13T13:51:09.874 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 154 (ObjNum 129 snap 0 seq_num 1540262200) clean dne 2024-09-13T13:51:09.875 INFO:tasks.rados.rados.0.smithi159.stdout:1304: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-13T13:51:09.882 INFO:tasks.rados.rados.0.smithi159.stdout:1304: writing smithi15938319-25 from 665610 to 1463470 tid 1 2024-09-13T13:51:09.883 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:09.882+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:09.887 INFO:tasks.rados.rados.0.smithi159.stdout:1304: writing smithi15938319-25 from 1909329 to 2367734 tid 2 2024-09-13T13:51:09.892 INFO:tasks.rados.rados.0.smithi159.stdout:1304: writing smithi15938319-25 from 2816700 to 3226910 tid 3 2024-09-13T13:51:09.892 INFO:tasks.rados.rados.0.smithi159.stdout:1299: done (5 left) 2024-09-13T13:51:09.892 INFO:tasks.rados.rados.0.smithi159.stdout:1300: done (4 left) 2024-09-13T13:51:09.893 INFO:tasks.rados.rados.0.smithi159.stdout:1305: write oid 11 current snap is 25 2024-09-13T13:51:09.893 INFO:tasks.rados.rados.0.smithi159.stdout:1301: finishing write tid 1 to smithi15938319-10 2024-09-13T13:51:09.893 INFO:tasks.rados.rados.0.smithi159.stdout:1301: finishing write tid 2 to smithi15938319-10 2024-09-13T13:51:09.893 INFO:tasks.rados.rados.0.smithi159.stdout:1305: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-13T13:51:09.898 INFO:tasks.rados.rados.0.smithi159.stdout:1305: writing smithi15938319-11 from 682417 to 1155526 tid 1 2024-09-13T13:51:09.904 INFO:tasks.rados.rados.0.smithi159.stdout:1305: writing smithi15938319-11 from 1567811 to 2169874 tid 2 2024-09-13T13:51:09.912 INFO:tasks.rados.rados.0.smithi159.stdout:1305: writing smithi15938319-11 from 2844123 to 3465605 tid 3 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1306: read oid 48 snap -1 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1306: expect (ObjNum 219 snap 24 seq_num 219) 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1307: snap_create 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: finishing write tid 3 to smithi15938319-10 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: finishing write tid 4 to smithi15938319-10 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: finishing write tid 5 to smithi15938319-10 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: oid 10 updating version 0 to 193 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: oid 10 updating version 193 to 194 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: oid 10 version 194 is already newer than 191 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: oid 10 version 194 is already newer than 192 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 194 (ObjNum 227 snap 25 seq_num 227) dirty exists 2024-09-13T13:51:09.913 INFO:tasks.rados.rados.0.smithi159.stdout:1301: left oid 10 (ObjNum 227 snap 25 seq_num 227) 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: finishing write tid 1 to smithi15938319-19 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: finishing write tid 2 to smithi15938319-19 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: finishing write tid 3 to smithi15938319-19 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: finishing write tid 4 to smithi15938319-19 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: finishing write tid 5 to smithi15938319-19 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: oid 19 updating version 0 to 102 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: oid 19 updating version 102 to 103 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: oid 19 updating version 103 to 105 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: oid 19 version 105 is already newer than 104 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 105 (ObjNum 228 snap 25 seq_num 228) dirty exists 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1302: left oid 19 (ObjNum 228 snap 25 seq_num 228) 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1303: finishing write tid 1 to smithi15938319-3 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1303: finishing write tid 2 to smithi15938319-3 2024-09-13T13:51:09.914 INFO:tasks.rados.rados.0.smithi159.stdout:1303: finishing write tid 3 to smithi15938319-3 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: finishing write tid 4 to smithi15938319-3 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: finishing write tid 5 to smithi15938319-3 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: oid 3 updating version 0 to 167 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: oid 3 version 167 is already newer than 166 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: oid 3 updating version 167 to 168 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: oid 3 updating version 168 to 169 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 169 (ObjNum 229 snap 25 seq_num 229) dirty exists 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1303: left oid 3 (ObjNum 229 snap 25 seq_num 229) 2024-09-13T13:51:09.915 INFO:tasks.rados.rados.0.smithi159.stdout:1304: finishing write tid 1 to smithi15938319-25 2024-09-13T13:51:09.916 INFO:tasks.rados.rados.0.smithi159.stdout:1304: finishing write tid 2 to smithi15938319-25 2024-09-13T13:51:09.916 INFO:tasks.rados.rados.0.smithi159.stdout:1304: finishing write tid 3 to smithi15938319-25 2024-09-13T13:51:09.916 INFO:tasks.rados.rados.0.smithi159.stdout:1305: finishing write tid 1 to smithi15938319-11 2024-09-13T13:51:09.916 INFO:tasks.rados.rados.0.smithi159.stdout:1305: finishing write tid 2 to smithi15938319-11 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: finishing write tid 3 to smithi15938319-11 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: finishing write tid 4 to smithi15938319-11 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: finishing write tid 5 to smithi15938319-11 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: oid 11 updating version 0 to 150 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: oid 11 updating version 150 to 151 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: oid 11 version 151 is already newer than 149 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: oid 11 updating version 151 to 152 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 152 (ObjNum 231 snap 25 seq_num 231) dirty exists 2024-09-13T13:51:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1305: left oid 11 (ObjNum 231 snap 25 seq_num 231) 2024-09-13T13:51:09.922 INFO:tasks.rados.rados.0.smithi159.stdout:1306: expect (ObjNum 219 snap 24 seq_num 219) 2024-09-13T13:51:09.963 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2671057 2024-09-13T13:51:09.983 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:09.983+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:09.988 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:09.988 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:09.997 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:09.999 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1517984 2024-09-13T13:51:10.085 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:10.084+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:10.102 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:10.102 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:10.111 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: finishing write tid 4 to smithi15938319-25 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: finishing write tid 5 to smithi15938319-25 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: oid 25 updating version 0 to 110 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: oid 25 version 110 is already newer than 109 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: oid 25 version 110 is already newer than 108 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:1304: oid 25 version 110 is already newer than 107 2024-09-13T13:51:10.123 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 110 (ObjNum 230 snap 25 seq_num 230) dirty exists 2024-09-13T13:51:10.124 INFO:tasks.rados.rados.0.smithi159.stdout:1304: left oid 25 (ObjNum 230 snap 25 seq_num 230) 2024-09-13T13:51:10.186 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:10.185+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:10.214 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:10.215 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:10.223 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:10.269 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:10.269+0000 7fe857f4f640 -1 osd.7 92 handle_osd_map injecting map crc failure 2024-09-13T13:51:10.287 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:10.286+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:10.367 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:10.376 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:10.418 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:10.416+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:10.491 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:10.489+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:10.519 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:10.520 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:10.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:10.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:10.590 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:10.589+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:10.650 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:10.650 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:10.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:10.705 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:10.705+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:10.715 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:10.736 INFO:tasks.rados.rados.0.smithi159.stdout:1301: done (6 left) 2024-09-13T13:51:10.736 INFO:tasks.rados.rados.0.smithi159.stdout:1302: done (5 left) 2024-09-13T13:51:10.736 INFO:tasks.rados.rados.0.smithi159.stdout:1303: done (4 left) 2024-09-13T13:51:10.736 INFO:tasks.rados.rados.0.smithi159.stdout:1304: done (3 left) 2024-09-13T13:51:10.737 INFO:tasks.rados.rados.0.smithi159.stdout:1305: done (2 left) 2024-09-13T13:51:10.737 INFO:tasks.rados.rados.0.smithi159.stdout:1306: done (1 left) 2024-09-13T13:51:10.737 INFO:tasks.rados.rados.0.smithi159.stdout:1307: done (0 left) 2024-09-13T13:51:10.737 INFO:tasks.rados.rados.0.smithi159.stdout:1308: set_chunk oid 30 target oid 2024-09-13T13:51:10.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:10.738 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:10.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:51:10.753 INFO:tasks.rados.rados.0.smithi159.stdout:1308: set_chunk oid 30 offset: 454144 length: 2093568 target oid 60a79308daf61e5ff0957c8a8204ef855e702800a9c0f603f9ad7a7f8910d49d offset: 0 snap -1 2024-09-13T13:51:10.753 INFO:tasks.rados.rados.0.smithi159.stdout:1309: read oid 42 snap -1 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1309: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1310: read oid 6 snap -1 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1310: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1311: read oid 19 snap -1 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1311: expect (ObjNum 228 snap 25 seq_num 228) 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1312: read oid 32 snap 16 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1312: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1313: read oid 8 snap -1 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1313: expect deleted 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1314: write (excl) oid 17 current snap is 26 2024-09-13T13:51:10.754 INFO:tasks.rados.rados.0.smithi159.stdout:1314: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-13T13:51:10.761 INFO:tasks.rados.rados.0.smithi159.stdout:1314: writing smithi15938319-17 from 699224 to 1247582 tid 1 2024-09-13T13:51:10.765 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:10.765 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:10.769 INFO:tasks.rados.rados.0.smithi159.stdout:1314: writing smithi15938319-17 from 2026293 to 2488367 tid 2 2024-09-13T13:51:10.776 INFO:tasks.rados.rados.0.smithi159.stdout:1314: writing smithi15938319-17 from 3271546 to 3704300 tid 3 2024-09-13T13:51:10.776 INFO:tasks.rados.rados.0.smithi159.stdout:1315: read oid 26 snap -1 2024-09-13T13:51:10.776 INFO:tasks.rados.rados.0.smithi159.stdout:1315: expect deleted 2024-09-13T13:51:10.776 INFO:tasks.rados.rados.0.smithi159.stdout:1316: write oid 24 current snap is 26 2024-09-13T13:51:10.776 INFO:tasks.rados.rados.0.smithi159.stdout:1316: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-13T13:51:10.779 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:10.779 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:10.779 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:10.779 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:10.779 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:10.783 INFO:tasks.rados.rados.0.smithi159.stdout:1316: writing smithi15938319-24 from 716031 to 1339638 tid 1 2024-09-13T13:51:10.790 INFO:tasks.rados.rados.0.smithi159.stdout:1316: writing smithi15938319-24 from 2084775 to 2690507 tid 2 2024-09-13T13:51:10.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:10.793 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:10.790+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:10.796 INFO:tasks.rados.rados.0.smithi159.stdout:1316: writing smithi15938319-24 from 3298969 to 3916031 tid 3 2024-09-13T13:51:10.797 INFO:tasks.rados.rados.0.smithi159.stdout:1317: write (excl) oid 7 current snap is 26 2024-09-13T13:51:10.797 INFO:tasks.rados.rados.0.smithi159.stdout:1317: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-13T13:51:10.800 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:10.800 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:10.804 INFO:tasks.rados.rados.0.smithi159.stdout:1317: writing smithi15938319-7 from 732838 to 1431694 tid 1 2024-09-13T13:51:10.810 INFO:tasks.rados.rados.0.smithi159.stdout:1317: writing smithi15938319-7 from 2026904 to 2492647 tid 2 2024-09-13T13:51:10.817 INFO:tasks.rados.rados.0.smithi159.stdout:1317: writing smithi15938319-7 from 3210039 to 3781690 tid 3 2024-09-13T13:51:10.817 INFO:tasks.rados.rados.0.smithi159.stdout:1318: read oid 9 snap -1 2024-09-13T13:51:10.817 INFO:tasks.rados.rados.0.smithi159.stdout:1318: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:10.817 INFO:tasks.rados.rados.0.smithi159.stdout:1319: set_chunk oid 44 target oid 2024-09-13T13:51:10.822 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:10.822 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:10.834 INFO:tasks.rados.rados.0.smithi159.stdout:1319: set_chunk oid 44 offset: 144896 length: 1482240 target oid 7804d680777f13332db5bd37b7bb866dce764018bbf6df3a3c7d3a17634be4c3 offset: 0 snap -1 2024-09-13T13:51:10.834 INFO:tasks.rados.rados.0.smithi159.stdout:1320: read oid 2 snap -1 2024-09-13T13:51:10.834 INFO:tasks.rados.rados.0.smithi159.stdout:1320: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:10.835 INFO:tasks.rados.rados.0.smithi159.stdout:1321: set_chunk oid 45 target oid 2024-09-13T13:51:10.842 INFO:tasks.rados.rados.0.smithi159.stdout:1321: set_chunk oid 45 offset: 2658816 length: 747008 target oid 6a1d026f45204edcc10490d0d06d7dd1880da2efc85cf3dfaa54371b16a9e659 offset: 0 snap 18 2024-09-13T13:51:10.842 INFO:tasks.rados.rados.0.smithi159.stdout:1322: set_chunk oid 27 target oid 2024-09-13T13:51:10.861 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:10.861 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:10.868 INFO:tasks.rados.rados.0.smithi159.stdout:1322: set_chunk oid 27 offset: 693760 length: 2681856 target oid 223f17e03198cc5ff445cd7caefb6c6fe1581ec9f29662a11751acb7e24edec5 offset: 0 snap -1 2024-09-13T13:51:10.868 INFO:tasks.rados.rados.0.smithi159.stdout:1323: delete oid 33 current snap is 26 2024-09-13T13:51:10.868 INFO:tasks.rados.rados.0.smithi159.stdout:1312: expect (ObjNum 161 snap 13 seq_num 161) 2024-09-13T13:51:10.868 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:10.893 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:10.892+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:10.897 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:10.897 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:10.919 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:10.919 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:10.941 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1281464 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 46.588775019000003, 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:51:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:51:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:51:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:51:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:51:10.950 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 45.949322838000001, 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:51:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 45.946725880999999, 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:51:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:51:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:51:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 42.207016138, 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:51:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 42.193571413000001, 2024-09-13T13:51:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:51:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:51:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.298624700000001, 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:51:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.254909615999999, 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:51:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.061096032999998, 2024-09-13T13:51:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:51:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 19.18070419, 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:51:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.965612845000001, 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:51:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:51:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:51:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.964734416000001, 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:51:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:51:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.941018917999999, 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:51:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 14.167656869, 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:51:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 14.167055980000001, 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:51:11.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 14.159903402999999, 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.014 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 13.854138988999999, 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.019 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:51:11.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:51:11.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 13.846507457, 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:11.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:51:11.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.135036314000001, 2024-09-13T13:51:11.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:11.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:51:11.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:51:11.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.033 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.134101347, 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:11.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:11.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.038 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.123418513000001, 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:11.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:51:11.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:51:11.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.044 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.044 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.045 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:11.045 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:11.045 INFO:tasks.rados.rados.0.smithi159.stdout:1308: finishing set_chunk to oid 30 2024-09-13T13:51:11.045 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 155 (ObjNum 215 snap 24 seq_num 215) dirty exists 2024-09-13T13:51:11.045 INFO:tasks.rados.rados.0.smithi159.stdout:1309: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:11.046 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:11.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:11.047 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:11.047 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:11.048 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:11.048 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:11.048 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:11.049 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:11.045+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:11.088 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:11.087+0000 7fc16a180640 -1 osd.11 93 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.146 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:11.146+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:11.176 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1958894 2024-09-13T13:51:11.194 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:11.194 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:11.195 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:11.195 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:11.204 INFO:tasks.rados.rados.0.smithi159.stdout:1311: expect (ObjNum 228 snap 25 seq_num 228) 2024-09-13T13:51:11.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:11.242 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:11.247 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:11.247+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:11.264 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:11.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:11.264+0000 7f3d0486f640 0 -- 172.21.15.27:0/1862551298 >> v2:172.21.15.122:6808/1255553857 conn(0x7f3d00221a20 msgr2=0x7f3d00223e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:51:11.311 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:11.311 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:11.311 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:11.312 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:11.312 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:11.348 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:11.347+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:11.449 INFO:tasks.rados.rados.0.smithi159.stdout:1310: expect (ObjNum 182 snap 17 seq_num 182) 2024-09-13T13:51:11.450 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:11.449+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:11.469 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:200 4.e 4:709ad093:::smithi15938319-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.734937645000002, 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075516929999999999, 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 200 2024-09-13T13:51:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732826+0000", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2300000000000002e-07 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732828+0000", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.328e-06 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732833+0000", 2024-09-13T13:51:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7269999999999998e-06 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732838+0000", 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2460000000000003e-06 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734037+0000", 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00119889 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734078+0000", 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1675000000000003e-05 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740269+0000", 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00619018 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740318+0000", 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9521999999999998e-05 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740377+0000", 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8801999999999997e-05 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:217 3.1 e41/33 3:90dfe591:::smithi15938319-18 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.683770756000001, 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097209170000000008, 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783993+0000", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787823+0000", 2024-09-13T13:51:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038302620000000001 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787880+0000", 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6681999999999997e-05 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787888+0000", 2024-09-13T13:51:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6270000000000006e-06 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787940+0000", 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2132000000000002e-05 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787966+0000", 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5741999999999998e-05 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793657+0000", 2024-09-13T13:51:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056903739999999998 2024-09-13T13:51:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793713+0000", 2024-09-13T13:51:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6449000000000001e-05 2024-09-13T13:51:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:223 3.1 e41/33 3:932c05fb:::smithi15938319-24 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.679932522000001, 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009035576, 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787831+0000", 2024-09-13T13:51:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790933+0000", 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031022459999999999 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791063+0000", 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129374 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791071+0000", 2024-09-13T13:51:11.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0020000000000006e-06 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791120+0000", 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9348999999999998e-05 2024-09-13T13:51:11.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791135+0000", 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5108e-05 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796837+0000", 2024-09-13T13:51:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057013999999999997 2024-09-13T13:51:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796866+0000", 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9295999999999999e-05 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:228 3.1 e41/33 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'5, mlcod=0'0)", 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.669984384000003, 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011341255999999999, 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797780+0000", 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.283e-06 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800442+0000", 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026622740000000001 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800472+0000", 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9765999999999999e-05 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800477+0000", 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2429999999999996e-06 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800515+0000", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7750000000000003e-05 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800533+0000", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8657000000000001e-05 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809070+0000", 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085367659999999995 2024-09-13T13:51:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809120+0000", 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9517e-05 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:220 3.1 e41/33 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'3)", 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.663617436999999, 2024-09-13T13:51:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015696446999999999, 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804147+0000", 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1629999999999999e-06 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810603+0000", 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064567410000000002 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810700+0000", 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6526000000000004e-05 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810708+0000", 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9009999999999994e-06 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810749+0000", 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810767+0000", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7798000000000001e-05 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819801+0000", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090344989999999997 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819842+0000", 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.053e-05 2024-09-13T13:51:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:231 3.1 e41/33 3:9f3f9b00:::smithi15938319-32 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.657151138000003, 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012065741, 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810613+0000", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8400000000000002e-07 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813879+0000", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003266658 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813938+0000", 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8579000000000003e-05 2024-09-13T13:51:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813945+0000", 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3540000000000002e-06 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813991+0000", 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5203000000000001e-05 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.814007+0000", 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6850999999999998e-05 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822644+0000", 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086365629999999999 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822677+0000", 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3549e-05 2024-09-13T13:51:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:232 3.1 e41/33 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.653878495999997, 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088135090000000006, 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813885+0000", 2024-09-13T13:51:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6899999999999997e-07 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817413+0000", 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035275020000000001 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817568+0000", 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155131 2024-09-13T13:51:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817574+0000", 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4479999999999999e-06 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817639+0000", 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5221000000000005e-05 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817657+0000", 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7450000000000001e-05 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822683+0000", 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050262070000000004 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822698+0000", 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4881e-05 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:235 3.1 e41/33 3:9761ca94:::smithi15938319-36 low_tier:head v 41'9, mlcod=41'4)", 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.650343468000003, 2024-09-13T13:51:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090169459999999996, 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817420+0000", 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2300000000000001e-07 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821383+0000", 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039627890000000004 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821417+0000", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4496999999999999e-05 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821424+0000", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0319999999999998e-06 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821470+0000", 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5748000000000003e-05 2024-09-13T13:51:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821488+0000", 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8207e-05 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826398+0000", 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004909409 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826436+0000", 2024-09-13T13:51:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8640999999999998e-05 2024-09-13T13:51:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:242 3.1 e41/33 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'8)", 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.636278673, 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083574370000000005, 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831485+0000", 2024-09-13T13:51:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2099999999999995e-07 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834193+0000", 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027077939999999999 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834220+0000", 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7018000000000001e-05 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834229+0000", 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7849999999999997e-06 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834275+0000", 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5961000000000003e-05 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834296+0000", 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1401999999999999e-05 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839806+0000", 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055102500000000004 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839842+0000", 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5305999999999997e-05 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:258 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.569579535000003, 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011133136, 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898185+0000", 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7400000000000001e-06 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898191+0000", 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898205+0000", 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4628e-05 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898213+0000", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5020000000000003e-06 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898410+0000", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000197235 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898490+0000", 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9856000000000003e-05 2024-09-13T13:51:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.904838+0000", 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063482139999999996 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.905236+0000", 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039744999999999998 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909250+0000", 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040139110000000002 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909281+0000", 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078999999999998e-05 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909316+0000", 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5778999999999997e-05 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:248 3.1 e41/33 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'13, mlcod=41'11)", 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.560564739, 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012034909999999999, 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907199+0000", 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4499999999999996e-07 2024-09-13T13:51:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911325+0000", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041262089999999996 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911350+0000", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4960000000000002e-05 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911356+0000", 2024-09-13T13:51:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4770000000000002e-06 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911406+0000", 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0633000000000001e-05 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911422+0000", 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5937999999999999e-05 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919153+0000", 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077307970000000002 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919233+0000", 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0050999999999996e-05 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:299 4.e 4:709ad093:::smithi15938319-50:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.698324145000001, 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089831609999999999, 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 299 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769441+0000", 2024-09-13T13:51:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5830000000000001e-06 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769453+0000", 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1756000000000001e-05 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769498+0000", 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4468999999999997e-05 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769510+0000", 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7052000000000005e-05 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769700+0000", 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2683999999999996e-05 2024-09-13T13:51:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777412+0000", 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077117100000000001 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777907+0000", 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049540900000000004 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778307+0000", 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039943500000000001 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778372+0000", 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5153000000000005e-05 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778422+0000", 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9941000000000002e-05 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:358 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 57.393299835000001, 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086044410000000009, 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 358 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074464+0000", 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074474+0000", 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5670000000000001e-06 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074488+0000", 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3889e-05 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074497+0000", 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:51:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074595+0000", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7899999999999994e-05 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074675+0000", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9849000000000001e-05 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077944+0000", 2024-09-13T13:51:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003269044 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082062+0000", 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041176290000000003 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082985+0000", 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092279199999999997 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083021+0000", 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6386e-05 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083067+0000", 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6483999999999997e-05 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:807 4.e 4:717ade62:::smithi15938319-24:head [write 541192~628415,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 53.286945764999999, 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010708927, 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 807 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180820+0000", 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7879999999999998e-06 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181481+0000", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066089999999999996 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181501+0000", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0503e-05 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181511+0000", 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9329999999999996e-06 2024-09-13T13:51:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181582+0000", 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0916999999999999e-05 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181588+0000", 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1160000000000004e-06 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185876+0000", 2024-09-13T13:51:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042876900000000003 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185941+0000", 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5168999999999999e-05 2024-09-13T13:51:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.186064+0000", 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012253599999999999 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191138+0000", 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050743660000000003 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191450+0000", 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031216700000000002 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191481+0000", 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0954000000000002e-05 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191526+0000", 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4888000000000001e-05 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1550 4.e 4:721f6cf4:::smithi15938319-18:6 [read 1158216~2203151,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.385760740000002, 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090284739999999999, 2024-09-13T13:51:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1550 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082003+0000", 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082013+0000", 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1330000000000005e-06 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082033+0000", 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9996e-05 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082048+0000", 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5593000000000001e-05 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088784+0000", 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067358349999999999 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088911+0000", 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126907 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.091031+0000", 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021196560000000001 2024-09-13T13:51:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1552 3.1 e59/33 3:bd95eec2:::8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338:head v 59'84, mlcod=59'82)", 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.374981921, 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081256560000000002, 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092783+0000", 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6929999999999999e-06 2024-09-13T13:51:11.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096023+0000", 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032406370000000002 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096060+0000", 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7160000000000003e-05 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096070+0000", 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5540000000000003e-06 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096139+0000", 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9385000000000001e-05 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096161+0000", 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1483000000000001e-05 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100843+0000", 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046821320000000003 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100907+0000", 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3612e-05 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1583 4.e 4:721f6cf4:::smithi15938319-18:head [read 0~3361367,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.008738964000003, 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00774357, 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1583 2024-09-13T13:51:11.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.29e-07 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459029+0000", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7890000000000002e-06 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459033+0000", 2024-09-13T13:51:11.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7409999999999998e-06 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459036+0000", 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2279999999999998e-06 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464333+0000", 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052973910000000003 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464381+0000", 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7741000000000003e-05 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.466767+0000", 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023864509999999999 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.571 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1598 4.e 4:717ade62:::smithi15938319-24:head [write 699191~532079] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.951914406, 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075604720000000004, 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1598 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515850+0000", 2024-09-13T13:51:11.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.449e-06 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516458+0000", 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060838899999999998 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516474+0000", 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5293000000000001e-05 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516481+0000", 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516553+0000", 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1365999999999994e-05 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516663+0000", 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109738 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516884+0000", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022128499999999999 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.519137+0000", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022529389999999998 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523336+0000", 2024-09-13T13:51:11.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041994830000000004 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523368+0000", 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1813000000000002e-05 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523409+0000", 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0812999999999997e-05 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2147 4.0 e74/34 4:04ca156e:::smithi15938319-49:head v 74'29, mlcod=74'26)", 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 20.661966948, 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.79346728, 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6899999999999997e-07 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805802+0000", 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7080000000000002e-06 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598451+0000", 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.79264855599999995 2024-09-13T13:51:11.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598460+0000", 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1279999999999997e-06 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598527+0000", 2024-09-13T13:51:11.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7605000000000001e-05 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598548+0000", 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0653999999999999e-05 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.599224+0000", 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00067555200000000005 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.599263+0000", 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9508000000000002e-05 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2313 4.e 4:717ade62:::smithi15938319-24:head [write 726102~788786,stat] snapc 13=[13,12,f,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:11.583 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.698307033000001, 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011702211000000001, 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2313 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:11.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769457+0000", 2024-09-13T13:51:11.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770373+0000", 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00091632300000000003 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770390+0000", 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6316000000000001e-05 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:11.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770408+0000", 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7907999999999999e-05 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770489+0000", 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1071999999999999e-05 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770495+0000", 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3280000000000002e-06 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773592+0000", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003096723 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773668+0000", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6488000000000004e-05 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773805+0000", 2024-09-13T13:51:11.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136291 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.776916+0000", 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003111688 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781046+0000", 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041299240000000001 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781111+0000", 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4973000000000003e-05 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781158+0000", 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6839999999999999e-05 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:11.590 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:11.592 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:11.593 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:11.593 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:11.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:11.594 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:11.591+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: finishing write tid 1 to smithi15938319-17 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: finishing write tid 2 to smithi15938319-17 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: finishing write tid 3 to smithi15938319-17 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: finishing write tid 4 to smithi15938319-17 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: finishing write tid 5 to smithi15938319-17 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: oid 17 updating version 0 to 196 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: oid 17 updating version 196 to 199 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: oid 17 version 199 is already newer than 198 2024-09-13T13:51:11.609 INFO:tasks.rados.rados.0.smithi159.stdout:1314: oid 17 version 199 is already newer than 197 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 199 (ObjNum 232 snap 26 seq_num 232) dirty exists 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1314: left oid 17 (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1316: finishing write tid 1 to smithi15938319-24 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1308: done (15 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1309: done (14 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1310: done (13 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1311: done (12 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1312: done (11 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1313: done (10 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1314: done (9 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1315: done (8 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1323: done (7 left) 2024-09-13T13:51:11.610 INFO:tasks.rados.rados.0.smithi159.stdout:1324: snap_remove snap 13 2024-09-13T13:51:11.692 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:11.691+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:11.740 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:11.741 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:11.741 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:11.741 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:11.793 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:11.793+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:11.851 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:51:11.852 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:11.852+0000 7f586c719640 -1 osd.9 94 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.854 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:11.853+0000 7faeac214640 -1 osd.8 94 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1324: done (7 left) 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1325: read oid 26 snap -1 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1325: expect deleted 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1326: read oid 17 snap -1 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1326: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1327: read oid 5 snap -1 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1327: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:1328: rollback oid 30 current snap is 26 2024-09-13T13:51:11.858 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 30 to 23 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: finishing write tid 2 to smithi15938319-24 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: finishing write tid 3 to smithi15938319-24 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: finishing write tid 4 to smithi15938319-24 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: finishing write tid 5 to smithi15938319-24 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: oid 24 updating version 0 to 169 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: oid 24 updating version 169 to 172 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: oid 24 version 172 is already newer than 170 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: oid 24 version 172 is already newer than 171 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 172 (ObjNum 233 snap 26 seq_num 233) dirty exists 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1316: left oid 24 (ObjNum 233 snap 26 seq_num 233) 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1317: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1317: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:11.859 INFO:tasks.rados.rados.0.smithi159.stdout:1317: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1316: done (10 left) 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1329: read oid 31 snap 21 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1329: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1330: delete oid 28 current snap is 26 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: finishing write tid 5 to smithi15938319-7 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: oid 7 updating version 0 to 172 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: oid 7 updating version 172 to 174 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: oid 7 version 174 is already newer than 173 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: oid 7 updating version 174 to 175 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 175 (ObjNum 234 snap 26 seq_num 234) dirty exists 2024-09-13T13:51:11.860 INFO:tasks.rados.rados.0.smithi159.stdout:1317: left oid 7 (ObjNum 234 snap 26 seq_num 234) 2024-09-13T13:51:11.861 INFO:tasks.rados.rados.0.smithi159.stdout:1318: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:11.861 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:11.861 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:11.861 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:11.862 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:11.862 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:11.862 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:11.857+0000 7f1a526ee640 -1 osd.13 94 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.862 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:11.855+0000 7fe0fbe07640 -1 osd.2 94 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.862 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:11.855+0000 7f8f5ac65640 -1 osd.6 94 handle_osd_map injecting map crc failure 2024-09-13T13:51:11.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:11.884 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2162434 2024-09-13T13:51:11.894 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:11.893+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:11.989 INFO:tasks.rados.rados.0.smithi159.stdout:1319: finishing set_chunk to oid 44 2024-09-13T13:51:11.989 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 147 (ObjNum 210 snap 22 seq_num 210) dirty exists 2024-09-13T13:51:11.989 INFO:tasks.rados.rados.0.smithi159.stdout:1320: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:11.992 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 617973 2024-09-13T13:51:11.995 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:11.995+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:12.020 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.852946006000003, 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:51:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.852920013000002, 2024-09-13T13:51:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:51:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:19 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.847974491999999, 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096180398, 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170583+0000", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170585+0000", 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1090000000000001e-06 2024-09-13T13:51:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266590+0000", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096005323000000004 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266598+0000", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8110000000000003e-06 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266650+0000", 2024-09-13T13:51:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2481000000000001e-05 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266658+0000", 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266763+0000", 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105209 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.747611190999997, 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:51:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:111 4.2 e41/34 4:4d382039:::smithi15938319-28:head v 41'9, mlcod=41'5)", 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.574571292000002, 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.06900017, 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1600000000000001e-07 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443991+0000", 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-09-13T13:51:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512563+0000", 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.068572151999999997 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512567+0000", 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7119999999999999e-06 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512607+0000", 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249000000000003e-05 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512623+0000", 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5866999999999999e-05 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512961+0000", 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033803199999999999 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512986+0000", 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5079000000000002e-05 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.451576013999997, 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:51:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:51:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.451561513999998, 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:51:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.449308074000001, 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:51:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.326884786000001, 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:51:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:51:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.326853966999998, 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:51:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:51:12.073 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.324695859000002, 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.074 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.075 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:51:12.076 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.115161661000002, 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.078 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:51:12.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:12.082 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.249184129, 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.083 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.086 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.235508414999998, 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.089 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.093 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.219606831, 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:51:12.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:51:12.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:51:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.211315336, 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.102 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:51:12.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:51:12.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:51:12.109 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.207288415000001, 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.110 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:51:12.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:51:12.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.117 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.206963459000001, 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:51:12.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:51:12.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:51:12.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:51:12.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 10.206931934, 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:51:12.125 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:51:12.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:51:12.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 7.1792419330000001, 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:51:12.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.136 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:12.138 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 264915 2024-09-13T13:51:12.138 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2995898 2024-09-13T13:51:12.138 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:12.137+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:12.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:12.140 INFO:tasks.rados.rados.0.smithi159.stdout:1321: finishing set_chunk to oid 45 2024-09-13T13:51:12.140 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 2658816 length: 747008 2024-09-13T13:51:12.140 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 86 snap 5 seq_num 86) dirty exists 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1322: finishing set_chunk to oid 27 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 164 (ObjNum 223 snap 24 seq_num 223) dirty exists 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1317: done (11 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1318: done (10 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1319: done (9 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1320: done (8 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1321: done (7 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1322: done (6 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1330: done (5 left) 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1331: read oid 10 snap 24 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1331: expect deleted 2024-09-13T13:51:12.141 INFO:tasks.rados.rados.0.smithi159.stdout:1332: set_chunk oid 22 target oid 2024-09-13T13:51:12.147 INFO:tasks.rados.rados.0.smithi159.stdout:1332: set_chunk oid 22 offset: 870912 length: 1125888 target oid b1e96245dadee09bcc3ae809b14c799f96fdd9a36796ed6ca5bc2439b94c05b5 offset: 0 snap 16 2024-09-13T13:51:12.147 INFO:tasks.rados.rados.0.smithi159.stdout:1333: write (excl) oid 2 current snap is 26 2024-09-13T13:51:12.147 INFO:tasks.rados.rados.0.smithi159.stdout:1333: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-13T13:51:12.154 INFO:tasks.rados.rados.0.smithi159.stdout:1333: writing smithi15938319-2 from 783259 to 1424215 tid 1 2024-09-13T13:51:12.160 INFO:tasks.rados.rados.0.smithi159.stdout:1333: writing smithi15938319-2 from 1918703 to 2531773 tid 2 2024-09-13T13:51:12.165 INFO:tasks.rados.rados.0.smithi159.stdout:1333: writing smithi15938319-2 from 3008661 to 3414128 tid 3 2024-09-13T13:51:12.168 INFO:tasks.rados.rados.0.smithi159.stdout:1333: writing smithi15938319-2 from 3821939 to 3983259 tid 4 2024-09-13T13:51:12.169 INFO:tasks.rados.rados.0.smithi159.stdout:1334: read oid 36 snap -1 2024-09-13T13:51:12.169 INFO:tasks.rados.rados.0.smithi159.stdout:1334: expect deleted 2024-09-13T13:51:12.169 INFO:tasks.rados.rados.0.smithi159.stdout:1335: read oid 40 snap 24 2024-09-13T13:51:12.169 INFO:tasks.rados.rados.0.smithi159.stdout:1335: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:12.169 INFO:tasks.rados.rados.0.smithi159.stdout:1336: write (excl) oid 8 current snap is 26 2024-09-13T13:51:12.170 INFO:tasks.rados.rados.0.smithi159.stdout:1336: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-13T13:51:12.174 INFO:tasks.rados.rados.0.smithi159.stdout:1336: writing smithi15938319-8 from 400066 to 1116271 tid 1 2024-09-13T13:51:12.179 INFO:tasks.rados.rados.0.smithi159.stdout:1336: writing smithi15938319-8 from 1860832 to 2000066 tid 2 2024-09-13T13:51:12.179 INFO:tasks.rados.rados.0.smithi159.stdout:1337: set_chunk oid 50 target oid 2024-09-13T13:51:12.179 INFO:tasks.rados.rados.0.smithi159.stdout:1337: set_chunk oid 50 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:12.180 INFO:tasks.rados.rados.0.smithi159.stdout:1338: read oid 33 snap 16 2024-09-13T13:51:12.180 INFO:tasks.rados.rados.0.smithi159.stdout:1338: expect (ObjNum 159 snap 13 seq_num 159) 2024-09-13T13:51:12.180 INFO:tasks.rados.rados.0.smithi159.stdout:1339: write (excl) oid 3 current snap is 26 2024-09-13T13:51:12.180 INFO:tasks.rados.rados.0.smithi159.stdout:1339: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-13T13:51:12.186 INFO:tasks.rados.rados.0.smithi159.stdout:1339: writing smithi15938319-3 from 416873 to 1208327 tid 1 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1339: writing smithi15938319-3 from 1919314 to 2016873 tid 2 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1340: read oid 38 snap -1 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1340: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1341: read oid 35 snap -1 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1341: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1328: finishing rollback tid 0 to smithi15938319-30 2024-09-13T13:51:12.192 INFO:tasks.rados.rados.0.smithi159.stdout:1329: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:51:12.205 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 941863 2024-09-13T13:51:12.239 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:12.239+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:12.250 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:12.250 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:12.250 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:12.250 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:12.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:12.294 INFO:tasks.rados.rados.0.smithi159.stdout:1328: finishing rollback tid 1 to smithi15938319-30 2024-09-13T13:51:12.294 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 179 (ObjNum 49 snap 0 seq_num 2) dirty dne 2024-09-13T13:51:12.294 INFO:tasks.rados.rados.0.smithi159.stdout:1327: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:12.340 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:12.339+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:12.369 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:12.370 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:12.370 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:12.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:12.370 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:12.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:12.425 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 675517 2024-09-13T13:51:12.440 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:12.440+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:12.478 INFO:tasks.rados.rados.0.smithi159.stdout:1326: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:12.483 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:12.484 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:12.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:12.484 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:12.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.484 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.583272962000002, 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.486 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:51:12.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.486 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:51:12.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:12.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:51:12.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:51:12.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.493 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.583205104999998, 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:51:12.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:51:12.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:264 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:12.499 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.569922179999999, 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010517171000000001, 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 264 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:12.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911504+0000", 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911506+0000", 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.598e-06 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911512+0000", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0669999999999997e-06 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911525+0000", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2567e-05 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911532+0000", 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6480000000000003e-06 2024-09-13T13:51:12.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911589+0000", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6773e-05 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911688+0000", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8844000000000004e-05 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921151+0000", 2024-09-13T13:51:12.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0094630770000000003 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921734+0000", 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058321100000000004 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921964+0000", 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000230144 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.921998+0000", 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3929e-05 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.922022+0000", 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3312999999999999e-05 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:364 4.c 4:3de93a23:::smithi15938319-15:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:12.505 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 58.406645511999997, 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010986604000000001, 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 364 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074781+0000", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074782+0000", 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074788+0000", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5230000000000001e-06 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074797+0000", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7609999999999996e-06 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074802+0000", 2024-09-13T13:51:12.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0549999999999999e-06 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074906+0000", 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000103646 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074954+0000", 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7939999999999998e-05 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080022+0000", 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050684679999999996 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.080619+0000", 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000596843 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085664+0000", 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050446689999999999 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085717+0000", 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.085768+0000", 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0803000000000001e-05 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1087 4.c 4:34c00adb:::smithi15938319-12:head [write 460525~690852] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:12.511 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 48.121147309999998, 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010491474000000001, 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1087 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360279+0000", 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.360281+0000", 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0999999999999998e-06 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364170+0000", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038886929999999999 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364189+0000", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9185999999999999e-05 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364198+0000", 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6819999999999992e-06 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364278+0000", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9839999999999995e-05 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364399+0000", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120767 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.364616+0000", 2024-09-13T13:51:12.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00021726100000000001 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.366707+0000", 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002091341 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370646+0000", 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039392430000000003 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370714+0000", 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8069000000000002e-05 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.370771+0000", 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6292000000000002e-05 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.517 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253230492999997, 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:51:12.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:12.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253196017999997, 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:12.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:51:12.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:51:12.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:12.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253181032000001, 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:51:12.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.532 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253166983, 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:51:12.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253145267999997, 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:51:12.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.253113423000002, 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:12.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:51:12.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091416697, 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:12.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:51:12.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091383547, 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.553 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:51:12.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:51:12.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091353329, 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:51:12.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091310980000003, 2024-09-13T13:51:12.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:51:12.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.569 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.572 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.572 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:12.572 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091276763000003, 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:51:12.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:51:12.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 44.091243491999997, 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:12.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:51:12.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:51:12.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.688960258999998, 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:51:12.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.6638936500000003, 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.589 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:51:12.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:51:12.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2587 4.c 4:34327e96:::smithi15938319-44:head [write 1772395~538068] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.6581306950000005, 2024-09-13T13:51:12.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012151192999999999, 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2587 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:12.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823297+0000", 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:12.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823897+0000", 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060006600000000001 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823910+0000", 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2243e-05 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823920+0000", 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0538e-05 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825222+0000", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001302194 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825304+0000", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1953999999999995e-05 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825429+0000", 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012415399999999999 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826790+0000", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013615050000000001 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835350+0000", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085598610000000002 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835406+0000", 2024-09-13T13:51:12.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5661000000000001e-05 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835447+0000", 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:12.601 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:12.602 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:12.603 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:12.604 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:12.602+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:12.617 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1332: finishing set_chunk to oid 22 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 1 to smithi15938319-2 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 2 to smithi15938319-2 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 3 to smithi15938319-2 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 4 to smithi15938319-2 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1325: done (15 left) 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1326: done (14 left) 2024-09-13T13:51:12.687 INFO:tasks.rados.rados.0.smithi159.stdout:1327: done (13 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1328: done (12 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1329: done (11 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1331: done (10 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1332: done (9 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1334: done (8 left) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1342: read oid 18 snap -1 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1342: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1343: write (excl) oid 16 current snap is 26 2024-09-13T13:51:12.688 INFO:tasks.rados.rados.0.smithi159.stdout:1335: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:12.698 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1406045 2024-09-13T13:51:12.703 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:12.702+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:12.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:12.773 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:12.804 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:12.803+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:12.826 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 5 to smithi15938319-2 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: finishing write tid 6 to smithi15938319-2 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: oid 2 updating version 0 to 163 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: oid 2 version 163 is already newer than 161 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: oid 2 version 163 is already newer than 160 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: oid 2 updating version 163 to 164 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: oid 2 version 164 is already newer than 162 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 164 (ObjNum 237 snap 26 seq_num 237) dirty exists 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1333: left oid 2 (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1336: finishing write tid 1 to smithi15938319-8 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1337: finishing set_chunk to oid 50 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1337: got expected ERANGE (src dne) 2024-09-13T13:51:12.827 INFO:tasks.rados.rados.0.smithi159.stdout:1336: finishing write tid 2 to smithi15938319-8 2024-09-13T13:51:12.828 INFO:tasks.rados.rados.0.smithi159.stdout:1336: finishing write tid 3 to smithi15938319-8 2024-09-13T13:51:12.828 INFO:tasks.rados.rados.0.smithi159.stdout:1343: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-13T13:51:12.831 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:12.831 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:12.831 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:12.831 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:12.831 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:12.832 INFO:tasks.rados.rados.0.smithi159.stdout:1343: writing smithi15938319-16 from 433680 to 900383 tid 1 2024-09-13T13:51:12.837 INFO:tasks.rados.rados.0.smithi159.stdout:1343: writing smithi15938319-16 from 1577796 to 2033680 tid 2 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1333: done (9 left) 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1335: done (8 left) 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1337: done (7 left) 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1344: snap_create 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1336: finishing write tid 4 to smithi15938319-8 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1336: oid 8 updating version 0 to 102 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1336: oid 8 version 102 is already newer than 101 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1336: oid 8 updating version 102 to 103 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 103 (ObjNum 238 snap 26 seq_num 238) dirty exists 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1336: left oid 8 (ObjNum 238 snap 26 seq_num 238) 2024-09-13T13:51:12.838 INFO:tasks.rados.rados.0.smithi159.stdout:1338: expect (ObjNum 159 snap 13 seq_num 159) 2024-09-13T13:51:12.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:12.849 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1292084 2024-09-13T13:51:12.855 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:12.853+0000 7f7c73c21640 -1 osd.14 95 handle_osd_map injecting map crc failure 2024-09-13T13:51:12.886 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:12.904 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:12.904+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:12.946 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 60.028267913999997, 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:51:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:51:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:51:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 60.023468385000001, 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:51:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:51:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:51:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:51:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.174120346999999, 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.869320035999998, 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:51:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:51:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:802 4.7 4:e1bd516b:::smithi15938319-19:head [write 524385~553166,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.780438791999998, 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074148269999999997, 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 802 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163120+0000", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163122+0000", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7680000000000001e-06 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163720+0000", 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00059817500000000005 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163735+0000", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5571999999999999e-05 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163748+0000", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2669e-05 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163797+0000", 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9388000000000002e-05 2024-09-13T13:51:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.163801+0000", 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1280000000000001e-06 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166043+0000", 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022420460000000001 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166150+0000", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010696600000000001 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.166368+0000", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021759899999999999 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170320+0000", 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039524 2024-09-13T13:51:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170454+0000", 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013373899999999999 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170491+0000", 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6405999999999997e-05 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.170535+0000", 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3971000000000003e-05 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.669822746999998, 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:51:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:51:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.663962798, 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:51:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:51:12.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:51:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.659018156999998, 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:51:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:51:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:51:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.658807578999998, 2024-09-13T13:51:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:51:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:51:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:51:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.658759480000001, 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:51:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:51:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.575679069000003, 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:51:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.575570358, 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:51:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:51:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.575455077999997, 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.035 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:51:13.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:51:13.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1328 4.7 4:e26a40dd:::smithi15938319-25:head [write 796665~544891] snapc 7=[7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 42.994055250000002, 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0085750689999999994, 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1328 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949503+0000", 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.949505+0000", 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1409999999999999e-06 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950074+0000", 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056815299999999995 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950091+0000", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7385999999999999e-05 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950118+0000", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7101e-05 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950198+0000", 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9474999999999994e-05 2024-09-13T13:51:13.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.950204+0000", 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4659999999999996e-06 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952911+0000", 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027069099999999999 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.952989+0000", 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8485000000000005e-05 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.953127+0000", 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013746799999999999 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957228+0000", 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041009810000000001 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.957981+0000", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075359999999999999 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958037+0000", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5349999999999997e-05 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.958078+0000", 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1553e-05 2024-09-13T13:51:13.046 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.701483975999999, 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.047 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:51:13.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.701397659000001, 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.701337289000001, 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:13.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.067485144000001, 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:51:13.062 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:51:13.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2633 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 1660464~1506637] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.1224145969999997, 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083516809999999997, 2024-09-13T13:51:13.069 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2633 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821144+0000", 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821147+0000", 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821153+0000", 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821158+0000", 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5820000000000001e-06 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821293+0000", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013434499999999999 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821298+0000", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.828410+0000", 2024-09-13T13:51:13.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071116349999999998 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.828495+0000", 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5327000000000005e-05 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829496+0000", 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001000754 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2634 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 0~2456909] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.1223871279999997, 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087606320000000008, 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.075 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2634 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821172+0000", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821175+0000", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5719999999999999e-06 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821180+0000", 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-13T13:51:13.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821185+0000", 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6590000000000004e-06 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821322+0000", 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013670500000000001 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821328+0000", 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8170000000000004e-06 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829522+0000", 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081938299999999992 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829600+0000", 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7750000000000006e-05 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829932+0000", 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033246000000000003 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.080 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: finishing write tid 1 to smithi15938319-3 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: finishing write tid 2 to smithi15938319-3 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: finishing write tid 3 to smithi15938319-3 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: finishing write tid 4 to smithi15938319-3 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: oid 3 updating version 0 to 189 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: oid 3 version 189 is already newer than 188 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: oid 3 version 189 is already newer than 187 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 189 (ObjNum 239 snap 26 seq_num 239) dirty exists 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1339: left oid 3 (ObjNum 239 snap 26 seq_num 239) 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:1340: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:13.082 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2435719 2024-09-13T13:51:13.083 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1049831 2024-09-13T13:51:13.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:13.084 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:13.080+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:13.184 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:13.182+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:13.186 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.186 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:13.186 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:13.186 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.196 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:13.209 INFO:tasks.rados.rados.0.smithi159.stdout:1341: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:13.283 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:13.282+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:13.297 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.298 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:13.298 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:13.298 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:13.298 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:13.339 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2891516 2024-09-13T13:51:13.384 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:13.384+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:13.408 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3302412 2024-09-13T13:51:13.410 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.410 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:13.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:13.410 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.639469570000003, 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:13.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:51:13.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:51:13.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:51:13.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:51:13.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:13.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:51:13.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.630439138, 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:51:13.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:51:13.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:51:13.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:51:13.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.429 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.978433262000003, 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.975399275999997, 2024-09-13T13:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.099567157999999, 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.441 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:51:13.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:51:13.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:51:13.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:51:13.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.093660325999998, 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:13.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:51:13.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:51:13.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.086224891000001, 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.454 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:51:13.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.459 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.166931149, 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:51:13.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.166862483999999, 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.467 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:51:13.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:51:13.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.166812370000002, 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:51:13.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.145855009000002, 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:51:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:51:13.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:51:13.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.485 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.140258805999999, 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:51:13.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:51:13.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.138714896000003, 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:13.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:51:13.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:51:13.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.138649848, 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.498 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:51:13.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:51:13.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.503 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 31.531708249000001, 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:51:13.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.676032545999998, 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:51:13.508 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:51:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.604136858, 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.514 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:51:13.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.519 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.604099337000001, 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:51:13.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:13.525 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.491180581999998, 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:13.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:51:13.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:51:13.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.277114174000001, 2024-09-13T13:51:13.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:51:13.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:51:13.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:51:13.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:13.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.539 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.541 INFO:tasks.rados.rados.0.smithi159.stdout:1342: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:13.541 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:13.540+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:13.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:13.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:13.640 INFO:tasks.rados.rados.0.smithi159.stdout:1343: finishing write tid 1 to smithi15938319-16 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1343: finishing write tid 2 to smithi15938319-16 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1336: done (7 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1338: done (6 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1339: done (5 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1340: done (4 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1341: done (3 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1342: done (2 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1344: done (1 left) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1345: read oid 47 snap -1 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1345: expect (ObjNum 225 snap 24 seq_num 225) 2024-09-13T13:51:13.641 INFO:tasks.rados.rados.0.smithi159.stdout:1346: write (excl) oid 49 current snap is 27 2024-09-13T13:51:13.642 INFO:tasks.rados.rados.0.smithi159.stdout:1343: finishing write tid 3 to smithi15938319-16 2024-09-13T13:51:13.642 INFO:tasks.rados.rados.0.smithi159.stdout:1346: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-13T13:51:13.642 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:13.642+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:13.644 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.644 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:13.644 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:13.644 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.645 INFO:tasks.rados.rados.0.smithi159.stdout:1346: writing smithi15938319-49 from 450487 to 992439 tid 1 2024-09-13T13:51:13.652 INFO:tasks.rados.rados.0.smithi159.stdout:1346: writing smithi15938319-49 from 1636278 to 2050487 tid 2 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1347: snap_create 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1343: finishing write tid 4 to smithi15938319-16 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1343: oid 16 updating version 0 to 209 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1343: oid 16 updating version 209 to 211 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1343: oid 16 version 211 is already newer than 210 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 211 (ObjNum 240 snap 26 seq_num 240) dirty exists 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1343: left oid 16 (ObjNum 240 snap 26 seq_num 240) 2024-09-13T13:51:13.653 INFO:tasks.rados.rados.0.smithi159.stdout:1346: finishing write tid 1 to smithi15938319-49 2024-09-13T13:51:13.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:13.657 INFO:tasks.rados.rados.0.smithi159.stdout:1346: finishing write tid 2 to smithi15938319-49 2024-09-13T13:51:13.658 INFO:tasks.rados.rados.0.smithi159.stdout:1346: finishing write tid 3 to smithi15938319-49 2024-09-13T13:51:13.658 INFO:tasks.rados.rados.0.smithi159.stdout:1346: finishing write tid 4 to smithi15938319-49 2024-09-13T13:51:13.658 INFO:tasks.rados.rados.0.smithi159.stdout:1346: oid 49 updating version 0 to 49 2024-09-13T13:51:13.659 INFO:tasks.rados.rados.0.smithi159.stdout:1346: oid 49 version 49 is already newer than 48 2024-09-13T13:51:13.659 INFO:tasks.rados.rados.0.smithi159.stdout:1346: oid 49 version 49 is already newer than 47 2024-09-13T13:51:13.659 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 49 (ObjNum 241 snap 27 seq_num 241) dirty exists 2024-09-13T13:51:13.659 INFO:tasks.rados.rados.0.smithi159.stdout:1346: left oid 49 (ObjNum 241 snap 27 seq_num 241) 2024-09-13T13:51:13.659 INFO:tasks.rados.rados.0.smithi159.stdout:1345: expect (ObjNum 225 snap 24 seq_num 225) 2024-09-13T13:51:13.729 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:13.728+0000 7fe859f53640 -1 osd.7 95 handle_osd_map injecting map crc failure 2024-09-13T13:51:13.743 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:13.743+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:13.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:13.748 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:13.758 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.758 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:13.758 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:13.758 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:13.758 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:13.770 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:13.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:13.794 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:13.814 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:13.835 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:13.837 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:13.844 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:13.843+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:13.856 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:13.856 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:13.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:13.911 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.676177026000005, 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:51:13.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 81.668142880000005, 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.916 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:51:13.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:51:13.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.920 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.272444720999999, 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:51:13.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.72554101900000001, 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:51:13.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:13.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.72070019500000004, 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:51:13.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:13.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.71998722000000004, 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:13.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:13.938 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:13.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:13.941 INFO:tasks.rados.rados.0.smithi159.stdout:1343: done (3 left) 2024-09-13T13:51:13.941 INFO:tasks.rados.rados.0.smithi159.stdout:1345: done (2 left) 2024-09-13T13:51:13.941 INFO:tasks.rados.rados.0.smithi159.stdout:1346: done (1 left) 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1347: done (0 left) 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1348: read oid 42 snap -1 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1348: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1349: write oid 18 current snap is 28 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1349: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1349: writing smithi15938319-18 from 467294 to 1084495 tid 1 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1349: writing smithi15938319-18 from 1578407 to 2067294 tid 2 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1350: read oid 13 snap -1 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1350: expect deleted 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1351: read oid 36 snap -1 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1351: expect deleted 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1352: read oid 35 snap -1 2024-09-13T13:51:13.942 INFO:tasks.rados.rados.0.smithi159.stdout:1352: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:13.943 INFO:tasks.rados.rados.0.smithi159.stdout:1353: snap_remove snap 23 2024-09-13T13:51:13.945 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:13.944+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:13.974 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:13.996 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:13.997 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:14.017 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:14.017 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:14.039 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:14.040 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:14.046 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:14.046+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:14.047 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:14.047 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:14.048 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:14.048 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:14.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:14.061 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:14.082 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:14.082 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:14.102 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:14.102 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:14.123 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:14.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:14.144+0000 7fe9fd532640 0 -- 172.21.15.27:0/914311976 >> v2:172.21.15.122:6808/1255553857 conn(0x7fe9f821fd10 msgr2=0x7fe9f8222100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:14.147 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:14.147+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:14.151 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:14.172 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.200 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.209 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:14.239 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:14.238+0000 7f002dbdb640 -1 osd.15 96 handle_osd_map injecting map crc failure 2024-09-13T13:51:14.247 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:14.247+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:14.348 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:14.348+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:14.351 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.351 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:14.449 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:14.448+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:14.475 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.475 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.484 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:14.507 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:14.506+0000 7f4ffbd7a640 -1 osd.1 96 handle_osd_map injecting map crc failure 2024-09-13T13:51:14.550 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:14.548+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:14.588 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.588 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:14.651 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:14.651+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:14.740 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.740 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.749 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:14.752 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:14.751+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:14.852 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.852 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.852 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:14.851+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:14.860 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:14.861 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:51:14.864 INFO:tasks.rados.rados.0.smithi159.stdout:1353: done (5 left) 2024-09-13T13:51:14.864 INFO:tasks.rados.rados.0.smithi159.stdout:1354: read oid 10 snap 27 2024-09-13T13:51:14.864 INFO:tasks.rados.rados.0.smithi159.stdout:1354: expect (ObjNum 227 snap 25 seq_num 227) 2024-09-13T13:51:14.865 INFO:tasks.rados.rados.0.smithi159.stdout:1355: write oid 26 current snap is 28 2024-09-13T13:51:14.865 INFO:tasks.rados.rados.0.smithi159.stdout:1348: expect (ObjNum 196 snap 18 seq_num 196) 2024-09-13T13:51:14.865 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:14.938 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1459502 2024-09-13T13:51:14.953 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:14.952+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:14.965 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:14.965 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:14.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:14.990 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1171311 2024-09-13T13:51:15.032 INFO:tasks.rados.rados.0.smithi159.stdout:1349: finishing write tid 1 to smithi15938319-18 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: finishing write tid 2 to smithi15938319-18 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: finishing write tid 3 to smithi15938319-18 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: finishing write tid 4 to smithi15938319-18 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: oid 18 updating version 0 to 182 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: oid 18 version 182 is already newer than 181 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: oid 18 version 182 is already newer than 180 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 182 (ObjNum 242 snap 28 seq_num 242) dirty exists 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1349: left oid 18 (ObjNum 242 snap 28 seq_num 242) 2024-09-13T13:51:15.033 INFO:tasks.rados.rados.0.smithi159.stdout:1352: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:15.054 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:15.052+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:15.080 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.080 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:15.173 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:15.171+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:15.198 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:15.232 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.232 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.241 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:15.256 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:15.255+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:15.281 INFO:tasks.rados.rados.0.smithi159.stdout:1354: expect (ObjNum 227 snap 25 seq_num 227) 2024-09-13T13:51:15.281 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:15.356 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:15.354+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:15.383 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.383 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:15.407 INFO:tasks.rados.rados.0.smithi159.stdout:1355: seq_num 243 ranges {484101=692450,1636889=447212} 2024-09-13T13:51:15.413 INFO:tasks.rados.rados.0.smithi159.stdout:1355: writing smithi15938319-26 from 484101 to 1176551 tid 1 2024-09-13T13:51:15.419 INFO:tasks.rados.rados.0.smithi159.stdout:1355: writing smithi15938319-26 from 1636889 to 2084101 tid 2 2024-09-13T13:51:15.419 INFO:tasks.rados.rados.0.smithi159.stdout:1348: done (6 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1349: done (5 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1350: done (4 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1351: done (3 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1352: done (2 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1354: done (1 left) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1356: read oid 29 snap -1 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1356: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1357: tier_evict oid 10 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1357: tier_evict oid 10 snap -1 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1355: finishing write tid 1 to smithi15938319-26 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1358: read oid 4 snap -1 2024-09-13T13:51:15.420 INFO:tasks.rados.rados.0.smithi159.stdout:1358: expect (ObjNum 214 snap 23 seq_num 214) 2024-09-13T13:51:15.421 INFO:tasks.rados.rados.0.smithi159.stdout:1359: set_chunk oid 14 target oid 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1359: set_chunk oid 14 offset: 390656 length: 849408 target oid e9371c13cd3e9738e01931ba1117eddaa24e121edeb9a68714c8b101ce8f86fe offset: 0 snap -1 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1360: read oid 38 snap -1 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1360: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1361: tier_flush oid 42 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1361: tier_flush oid 42 snap -1 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1355: finishing write tid 2 to smithi15938319-26 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1362: rollback oid 49 current snap is 28 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 49 to 20 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1355: finishing write tid 3 to smithi15938319-26 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1355: finishing write tid 4 to smithi15938319-26 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1355: oid 26 updating version 0 to 89 2024-09-13T13:51:15.434 INFO:tasks.rados.rados.0.smithi159.stdout:1355: oid 26 version 89 is already newer than 88 2024-09-13T13:51:15.435 INFO:tasks.rados.rados.0.smithi159.stdout:1355: oid 26 updating version 89 to 90 2024-09-13T13:51:15.435 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 90 (ObjNum 243 snap 28 seq_num 243) dirty exists 2024-09-13T13:51:15.435 INFO:tasks.rados.rados.0.smithi159.stdout:1355: left oid 26 (ObjNum 243 snap 28 seq_num 243) 2024-09-13T13:51:15.435 INFO:tasks.rados.rados.0.smithi159.stdout:1356: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:15.477 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:15.477+0000 7f9b0c553640 -1 osd.12 97 handle_osd_map injecting map crc failure 2024-09-13T13:51:15.478 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:15.477+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:15.487 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2817272 2024-09-13T13:51:15.539 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.539 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:15.548 INFO:tasks.rados.rados.0.smithi159.stdout:1357: got (22) Invalid argument 2024-09-13T13:51:15.548 INFO:tasks.rados.rados.0.smithi159.stdout:1355: done (7 left) 2024-09-13T13:51:15.548 INFO:tasks.rados.rados.0.smithi159.stdout:1356: done (6 left) 2024-09-13T13:51:15.549 INFO:tasks.rados.rados.0.smithi159.stdout:1357: done (5 left) 2024-09-13T13:51:15.549 INFO:tasks.rados.rados.0.smithi159.stdout:1363: snap_remove snap 22 2024-09-13T13:51:15.558 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:15.557+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:15.659 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:15.658+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:15.690 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.690 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:15.760 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:15.759+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:15.801 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.801 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:15.860 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:15.860+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:15.869 INFO:tasks.rados.rados.0.smithi159.stdout:1363: done (5 left) 2024-09-13T13:51:15.869 INFO:tasks.rados.rados.0.smithi159.stdout:1364: snap_remove snap 27 2024-09-13T13:51:15.917 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:15.917 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:15.926 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:15.962 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:15.961+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:16.032 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:16.032 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:16.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:16.062 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:16.062+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:16.148 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:16.148 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:16.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:16.162 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:16.161+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:16.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:16.259 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:16.259 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:16.263 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:16.263+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:16.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:16.332 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:16.332 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:16.355 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:16.356 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:16.364 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:16.364+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:16.370 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:16.370 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:16.377 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:16.378 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:16.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:16.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:16.441 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:16.441 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:16.462 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:16.462 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:16.464 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:16.464+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:16.481 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:16.481 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:16.483 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:16.483 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:16.490 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:16.504 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:16.526 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:16.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:16.547 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:16.566 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:16.565+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:16.568 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:16.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:16.591 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:16.612 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:16.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:16.631+0000 7fcd16ffd640 0 -- 172.21.15.27:0/1456061013 >> v2:172.21.15.122:6808/1255553857 conn(0x7fcd1821fa70 msgr2=0x7fcd18221e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:51:16.635 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:16.635 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:16.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:16.635 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:16.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.892323085999998, 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.636 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:51:16.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.640 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.886741827000002, 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:51:16.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:51:16.644 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.883765216, 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:51:16.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:51:16.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:16.650 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.881791045, 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:51:16.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:16.655 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.848031943999999, 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:16.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:51:16.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:51:16.660 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.843060522000002, 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.661 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:51:16.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:51:16.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.839159230999996, 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.666 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:51:16.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:51:16.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:238 3.0 e41/33 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'9)", 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.805747402999998, 2024-09-13T13:51:16.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076037040000000002, 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827890+0000", 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827891+0000", 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.248e-06 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828642+0000", 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075090499999999995 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828710+0000", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7478999999999995e-05 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828723+0000", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3499999999999999e-05 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828822+0000", 2024-09-13T13:51:16.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9061999999999994e-05 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.828853+0000", 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0939999999999999e-05 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835418+0000", 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065648920000000001 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835494+0000", 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5678000000000003e-05 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.799121163000002, 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.676 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:51:16.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.793945321000002, 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.681 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:51:16.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.717994243, 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:51:16.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.713193347999997, 2024-09-13T13:51:16.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:51:16.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:51:16.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:51:16.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.699 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.699 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:325 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.862531773000001, 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072630530000000002, 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 325 2024-09-13T13:51:16.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771106+0000", 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771107+0000", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.252e-06 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771114+0000", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0119999999999999e-06 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771129+0000", 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5131e-05 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771137+0000", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9570000000000002e-06 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771199+0000", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2007000000000004e-05 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.771272+0000", 2024-09-13T13:51:16.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3152999999999996e-05 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777134+0000", 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005861387 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778167+0000", 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010329270000000001 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778274+0000", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000106827 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778323+0000", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9941999999999997e-05 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778369+0000", 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5457999999999997e-05 2024-09-13T13:51:16.705 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.558554510999997, 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.706 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 58.435673358000003, 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:51:16.712 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:51:16.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:51:16.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.718 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.378693024999997, 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.719 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:51:16.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.373380175999998, 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.725 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:51:16.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:16.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.731 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.345601668999997, 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:16.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.735 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.14.0:186 4.2 e70/63 4:4dc7b2b1:::smithi15938319-14:c v 70'101, mlcod=69'92)", 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.340377031999999, 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0077348039999999996, 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293261+0000", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3099999999999998e-07 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293268+0000", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8480000000000003e-06 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293286+0000", 2024-09-13T13:51:16.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7802999999999999e-05 2024-09-13T13:51:16.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.293290+0000", 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1710000000000004e-06 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.295355+0000", 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020649639999999999 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.295371+0000", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5777000000000002e-05 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.300956+0000", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055851629999999998 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.300995+0000", 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9647000000000002e-05 2024-09-13T13:51:16.741 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 20.836961414000001, 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:16.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:16.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:51:16.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.746 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:16.747 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:16.747 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:16.748 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:16.748 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:16.746+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:16.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:16.750 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:16.849 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:16.849+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:16.854 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:16.854 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:16.855 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:16.855 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:16.865 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:16.873 INFO:tasks.rados.rados.0.smithi159.stdout:1364: done (5 left) 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1365: read oid 12 snap -1 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1365: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1366: rollback oid 41 current snap is 28 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 41 to 17 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1359: finishing set_chunk to oid 14 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk returned error code -95 offset: 390656 length: 849408 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 174 snap 16 seq_num 174) dirty exists 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1361: got (16) Device or resource busy 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 42 v 85 (ObjNum 196 snap 18 seq_num 196) unflushed 18 2024-09-13T13:51:16.874 INFO:tasks.rados.rados.0.smithi159.stdout:1358: expect (ObjNum 214 snap 23 seq_num 214) 2024-09-13T13:51:16.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:51:16.946 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2585550 2024-09-13T13:51:16.949 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:16.948+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:16.961 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:16.969 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:16.970 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:16.970 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:16.978 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:17.012 INFO:tasks.rados.rados.0.smithi159.stdout:1362: finishing rollback tid 0 to smithi15938319-49 2024-09-13T13:51:17.012 INFO:tasks.rados.rados.0.smithi159.stdout:1362: finishing rollback tid 1 to smithi15938319-49 2024-09-13T13:51:17.012 INFO:tasks.rados.rados.0.smithi159.stdout:1362: finishing rollback tid 2 to smithi15938319-49 2024-09-13T13:51:17.012 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 55 (ObjNum 173 snap 16 seq_num 173) dirty exists 2024-09-13T13:51:17.012 INFO:tasks.rados.rados.0.smithi159.stdout:1360: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:17.050 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:17.050+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.853724311999997, 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:51:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.852770813000006, 2024-09-13T13:51:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:51:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.626520838999994, 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:51:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:51:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 53.341934553000002, 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:51:17.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.896568772999998, 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.137 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:17.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:51:17.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2049 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'52, mlcod=66'48)", 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.791917386000001, 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003216062, 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.142 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323795+0000", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025235600000000002 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323833+0000", 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8479000000000002e-05 2024-09-13T13:51:17.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323843+0000", 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0142999999999999e-05 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323913+0000", 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9250000000000003e-05 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323940+0000", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7047999999999999e-05 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326715+0000", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027753679999999998 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326758+0000", 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3248999999999999e-05 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.146 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.78902398, 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:17.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:51:17.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:51:17.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:51:17.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.155 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:17.156 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.788275090999999, 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:17.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:51:17.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:51:17.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:51:17.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.166 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2123 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'60, mlcod=70'56)", 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 26.375740108999999, 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035624979999999999, 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.167 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3299999999999998e-07 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740298+0000", 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057840599999999995 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740767+0000", 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046913499999999997 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740784+0000", 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6487999999999999e-05 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740877+0000", 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2666999999999999e-05 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740903+0000", 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6177999999999999e-05 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743246+0000", 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023431390000000002 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743282+0000", 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6152000000000002e-05 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:51:17.172 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 26.369940909, 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:51:17.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 26.364812551, 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:51:17.177 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:51:17.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:51:17.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:17.182 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 26.363900995000002, 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:17.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:51:17.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.189 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:134 4.8 e78/34 4:1b24d731:::smithi15938319-13:11 v 78'71, mlcod=75'65)", 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.496691234, 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033973689999999999, 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:17.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618769+0000", 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618785+0000", 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6127999999999999e-05 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618814+0000", 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9110000000000001e-05 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618821+0000", 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3130000000000004e-06 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618874+0000", 2024-09-13T13:51:17.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3596e-05 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618890+0000", 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6203999999999999e-05 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622135+0000", 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032444000000000001 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622166+0000", 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0969000000000001e-05 2024-09-13T13:51:17.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 20.335814907, 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.197 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:138 4.8 e80/34 4:1b24d731:::smithi15938319-13:11 v 80'75, mlcod=78'69)", 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:17.202 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 20.335148582999999, 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038853730000000001, 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:17.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780312+0000", 2024-09-13T13:51:17.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:51:17.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780322+0000", 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0383999999999999e-05 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780367+0000", 2024-09-13T13:51:17.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4783000000000001e-05 2024-09-13T13:51:17.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780378+0000", 2024-09-13T13:51:17.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0292000000000001e-05 2024-09-13T13:51:17.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780701+0000", 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032313399999999999 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780722+0000", 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1353e-05 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784153+0000", 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003430986 2024-09-13T13:51:17.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784196+0000", 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3498999999999998e-05 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:144 4.8 e84/34 4:165e03eb:::smithi15938319-32:a v 84'77, mlcod=80'73)", 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.315823389999998, 2024-09-13T13:51:17.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032387330000000001, 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799637+0000", 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9299999999999999e-07 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799641+0000", 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2629999999999997e-06 2024-09-13T13:51:17.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800327+0000", 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068558599999999996 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800335+0000", 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800408+0000", 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3792000000000001e-05 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800431+0000", 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2883000000000001e-05 2024-09-13T13:51:17.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.802832+0000", 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024011869999999999 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.802875+0000", 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2663999999999997e-05 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.216 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2559 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'82, mlcod=84'79)", 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.269914612000001, 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040899279999999996, 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845546+0000", 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845704+0000", 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015839000000000001 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845732+0000", 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7736e-05 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845738+0000", 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845791+0000", 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845807+0000", 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5755000000000001e-05 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849592+0000", 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037853499999999998 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849635+0000", 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2669000000000001e-05 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:17.222 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.266540563, 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:51:17.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.265772515, 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.227 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:17.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:51:17.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.9094592349999999, 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.232 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:51:17.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.236 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.237 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.237 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:17.238 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:17.239 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:17.237+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:17.256 INFO:tasks.rados.rados.0.smithi159.stdout:1365: expect (ObjNum 152 snap 12 seq_num 152) 2024-09-13T13:51:17.270 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:17.268+0000 7f4ffdd7e640 -1 osd.1 99 handle_osd_map injecting map crc failure 2024-09-13T13:51:17.271 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1680030 2024-09-13T13:51:17.272 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:17.271+0000 7f586a715640 -1 osd.9 99 handle_osd_map injecting map crc failure 2024-09-13T13:51:17.338 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:17.338+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:17.344 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:17.344 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:17.344 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:17.353 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:17.407 INFO:tasks.rados.rados.0.smithi159.stdout:1366: finishing rollback tid 0 to smithi15938319-41 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1366: finishing rollback tid 1 to smithi15938319-41 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1366: finishing rollback tid 2 to smithi15938319-41 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 170 (ObjNum 119 snap 8 seq_num 119) dirty exists 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1358: done (6 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1359: done (5 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1360: done (4 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1361: done (3 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1362: done (2 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1365: done (1 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1366: done (0 left) 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1367: write (excl) oid 26 current snap is 28 2024-09-13T13:51:17.408 INFO:tasks.rados.rados.0.smithi159.stdout:1367: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-13T13:51:17.412 INFO:tasks.rados.rados.0.smithi159.stdout:1367: writing smithi15938319-26 from 500908 to 984960 tid 1 2024-09-13T13:51:17.419 INFO:tasks.rados.rados.0.smithi159.stdout:1367: writing smithi15938319-26 from 1411724 to 2100908 tid 2 2024-09-13T13:51:17.419 INFO:tasks.rados.rados.0.smithi159.stdout:1368: write oid 24 current snap is 28 2024-09-13T13:51:17.419 INFO:tasks.rados.rados.0.smithi159.stdout:1367: finishing write tid 1 to smithi15938319-26 2024-09-13T13:51:17.419 INFO:tasks.rados.rados.0.smithi159.stdout:1368: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-13T13:51:17.425 INFO:tasks.rados.rados.0.smithi159.stdout:1368: writing smithi15938319-24 from 517715 to 1077016 tid 1 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1368: writing smithi15938319-24 from 1870206 to 2117715 tid 2 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: finishing write tid 2 to smithi15938319-26 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: finishing write tid 3 to smithi15938319-26 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: finishing write tid 4 to smithi15938319-26 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: oid 26 updating version 0 to 92 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: oid 26 updating version 92 to 93 2024-09-13T13:51:17.431 INFO:tasks.rados.rados.0.smithi159.stdout:1367: oid 26 version 93 is already newer than 91 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 93 (ObjNum 244 snap 28 seq_num 244) dirty exists 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1367: left oid 26 (ObjNum 244 snap 28 seq_num 244) 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1368: finishing write tid 1 to smithi15938319-24 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1369: read oid 22 snap -1 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1369: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1367: done (2 left) 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1370: read oid 16 snap 17 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1370: expect (ObjNum 171 snap 16 seq_num 171) 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1371: write oid 13 current snap is 28 2024-09-13T13:51:17.432 INFO:tasks.rados.rados.0.smithi159.stdout:1371: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-13T13:51:17.437 INFO:tasks.rados.rados.0.smithi159.stdout:1371: writing smithi15938319-13 from 534522 to 1169072 tid 1 2024-09-13T13:51:17.439 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:17.438+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1371: writing smithi15938319-13 from 1812335 to 2134522 tid 2 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1372: read oid 7 snap -1 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1372: expect (ObjNum 234 snap 26 seq_num 234) 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1373: tier_evict oid 32 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1373: tier_evict oid 32 snap -1 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1374: tier_evict oid 18 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1374: tier_evict oid 18 snap -1 2024-09-13T13:51:17.443 INFO:tasks.rados.rados.0.smithi159.stdout:1369: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:51:17.513 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:17.513 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:17.513 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:17.513 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:17.513 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:17.519 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 991549 2024-09-13T13:51:17.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: finishing write tid 2 to smithi15938319-24 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: finishing write tid 3 to smithi15938319-24 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: finishing write tid 4 to smithi15938319-24 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: oid 24 updating version 0 to 194 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: oid 24 updating version 194 to 196 2024-09-13T13:51:17.538 INFO:tasks.rados.rados.0.smithi159.stdout:1368: oid 24 version 196 is already newer than 195 2024-09-13T13:51:17.539 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 196 (ObjNum 245 snap 28 seq_num 245) dirty exists 2024-09-13T13:51:17.539 INFO:tasks.rados.rados.0.smithi159.stdout:1368: left oid 24 (ObjNum 245 snap 28 seq_num 245) 2024-09-13T13:51:17.539 INFO:tasks.rados.rados.0.smithi159.stdout:1370: expect (ObjNum 171 snap 16 seq_num 171) 2024-09-13T13:51:17.540 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:17.540+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:17.560 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2379549 2024-09-13T13:51:17.641 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:17.640+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:17.643 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:17.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.099930659999998, 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:51:17.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:51:17.648 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.440488594000001, 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.439100370000006, 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.653 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:51:17.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.657 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.438398367000005, 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:51:17.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.661 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:96 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'63)", 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.437912564000001, 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050574289999999996, 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.662 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204033+0000", 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204133+0000", 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9450000000000005e-05 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204163+0000", 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0018999999999998e-05 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204168+0000", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5539999999999997e-06 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206261+0000", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020930889999999998 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206274+0000", 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3018e-05 2024-09-13T13:51:17.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209065+0000", 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027905909999999998 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209090+0000", 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4896999999999999e-05 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.044796030000001, 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:51:17.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:116 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'86, mlcod=25'80)", 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.044251591000005, 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051291219999999998, 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.671 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597694+0000", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597697+0000", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6440000000000002e-06 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597713+0000", 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5699999999999999e-05 2024-09-13T13:51:17.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597718+0000", 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1669999999999998e-06 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599712+0000", 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019937420000000002 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599730+0000", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8091e-05 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602796+0000", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003066306 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602823+0000", 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6112999999999999e-05 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.675 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.038672227000006, 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:51:17.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.679 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:17.680 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.725115071000005, 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:17.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:51:17.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:51:17.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.871276323000004, 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.687 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:51:17.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:17.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.692 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:812 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 2412870~427480,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.451926055999998, 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054419789999999996, 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.693 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 812 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190022+0000", 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190034+0000", 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190058+0000", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190071+0000", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3861e-05 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190156+0000", 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4179999999999997e-05 2024-09-13T13:51:17.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190164+0000", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194705+0000", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045405109999999997 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194761+0000", 2024-09-13T13:51:17.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6140999999999997e-05 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195461+0000", 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070009399999999996 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.451841453999997, 2024-09-13T13:51:17.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:51:17.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:17.704 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.451793504000001, 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:51:17.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:51:17.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 57.763330576999998, 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.710 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:51:17.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:51:17.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:51:17.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:17.716 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.067723450000003, 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:51:17.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 30.340021176, 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.723 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:51:17.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 30.314511770999999, 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.728 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:51:17.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 30.313846034000001, 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.733 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:51:17.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 30.313805018, 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.738 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:17.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.742 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 5.4946681100000001, 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:17.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:51:17.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:51:17.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:51:17.748 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.749 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:17.749 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:17.749 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1371: finishing write tid 1 to smithi15938319-13 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1374: got (22) Invalid argument 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1373: got (22) Invalid argument 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1371: finishing write tid 2 to smithi15938319-13 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1368: done (6 left) 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1369: done (5 left) 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1370: done (4 left) 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1373: done (3 left) 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1374: done (2 left) 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1375: read oid 28 snap -1 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1375: expect deleted 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1376: read oid 35 snap -1 2024-09-13T13:51:17.751 INFO:tasks.rados.rados.0.smithi159.stdout:1376: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1377: write oid 46 current snap is 28 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: finishing write tid 3 to smithi15938319-13 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: finishing write tid 4 to smithi15938319-13 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: oid 13 updating version 0 to 104 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: oid 13 updating version 104 to 105 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: oid 13 version 105 is already newer than 103 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 105 (ObjNum 246 snap 28 seq_num 246) dirty exists 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1371: left oid 13 (ObjNum 246 snap 28 seq_num 246) 2024-09-13T13:51:17.752 INFO:tasks.rados.rados.0.smithi159.stdout:1377: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-13T13:51:17.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:17.754 INFO:tasks.rados.rados.0.smithi159.stdout:1377: writing smithi15938319-46 from 551329 to 1261128 tid 1 2024-09-13T13:51:17.754 INFO:tasks.rados.rados.0.smithi159.stdout:1377: writing smithi15938319-46 from 1870817 to 2151329 tid 2 2024-09-13T13:51:17.754 INFO:tasks.rados.rados.0.smithi159.stdout:1371: done (4 left) 2024-09-13T13:51:17.754 INFO:tasks.rados.rados.0.smithi159.stdout:1378: write oid 39 current snap is 28 2024-09-13T13:51:17.754 INFO:tasks.rados.rados.0.smithi159.stdout:1372: expect (ObjNum 234 snap 26 seq_num 234) 2024-09-13T13:51:17.755 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:17.748+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:17.832 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3365266 2024-09-13T13:51:17.851 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:17.850+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:17.867 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:17.867 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:17.867 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:17.872 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:51:17.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:17.949 INFO:tasks.rados.rados.0.smithi159.stdout:1376: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:17.952 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:17.951+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:17.984 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:17.985 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:17.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:18.037 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 865907 2024-09-13T13:51:18.053 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:18.052+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:51:18.102 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 53.747180423000003, 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.106 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:18.107 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 53.107728242, 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 53.105131284999999, 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:18.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:18.115 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.365421542, 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:18.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:51:18.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.351976817000001, 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:51:18.119 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:51:18.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.457030103999998, 2024-09-13T13:51:18.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:18.127 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.413315019999999, 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:51:18.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:51:18.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 36.219501436999998, 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:51:18.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.136 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 26.339109594, 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:51:18.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:51:18.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.124018248999999, 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.141 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:51:18.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.123139819999999, 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:51:18.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:51:18.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.099424322000001, 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:51:18.150 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:51:18.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.326062273000002, 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:18.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:51:18.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:18.159 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.325461384, 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:51:18.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.318308807000001, 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.164 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:51:18.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:51:18.168 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.012544392999999, 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:18.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.173 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 21.004912861000001, 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:51:18.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:51:18.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.293441718, 2024-09-13T13:51:18.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:51:18.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:51:18.182 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.292506751000001, 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:51:18.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:51:18.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:51:18.187 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.281823917000001, 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:51:18.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.191 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.192 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.192 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.192 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: finishing write tid 1 to smithi15938319-46 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: finishing write tid 2 to smithi15938319-46 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: finishing write tid 3 to smithi15938319-46 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: finishing write tid 4 to smithi15938319-46 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: oid 46 updating version 0 to 94 2024-09-13T13:51:18.193 INFO:tasks.rados.rados.0.smithi159.stdout:1377: oid 46 updating version 94 to 96 2024-09-13T13:51:18.194 INFO:tasks.rados.rados.0.smithi159.stdout:1377: oid 46 version 96 is already newer than 95 2024-09-13T13:51:18.194 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 96 (ObjNum 247 snap 28 seq_num 247) dirty exists 2024-09-13T13:51:18.194 INFO:tasks.rados.rados.0.smithi159.stdout:1377: left oid 46 (ObjNum 247 snap 28 seq_num 247) 2024-09-13T13:51:18.194 INFO:tasks.rados.rados.0.smithi159.stdout:1378: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-13T13:51:18.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:18.195 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:18.192+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:18.198 INFO:tasks.rados.rados.0.smithi159.stdout:1378: writing smithi15938319-39 from 568136 to 1353184 tid 1 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1378: writing smithi15938319-39 from 1929299 to 2168136 tid 2 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1372: done (4 left) 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1375: done (3 left) 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1376: done (2 left) 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1377: done (1 left) 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1379: read oid 2 snap 25 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1379: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:18.203 INFO:tasks.rados.rados.0.smithi159.stdout:1380: read oid 25 snap -1 2024-09-13T13:51:18.204 INFO:tasks.rados.rados.0.smithi159.stdout:1380: expect (ObjNum 230 snap 25 seq_num 230) 2024-09-13T13:51:18.204 INFO:tasks.rados.rados.0.smithi159.stdout:1381: write (excl) oid 42 current snap is 28 2024-09-13T13:51:18.204 INFO:tasks.rados.rados.0.smithi159.stdout:1381: seq_num 249 ranges {584943=460297,1587781=597162} 2024-09-13T13:51:18.208 INFO:tasks.rados.rados.0.smithi159.stdout:1381: writing smithi15938319-42 from 584943 to 1045240 tid 1 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1381: writing smithi15938319-42 from 1587781 to 2184943 tid 2 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1382: read oid 35 snap -1 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1382: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1383: read oid 1 snap 18 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1383: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:18.215 INFO:tasks.rados.rados.0.smithi159.stdout:1384: set_chunk oid 6 target oid 2024-09-13T13:51:18.219 INFO:tasks.rados.rados.0.smithi159.stdout:1384: set_chunk oid 6 offset: 966656 length: 22528 target oid c3acbe5d654e07ce5280c1af192cbb39a12e0bc3fdbaeb27dfd1557a3f60e408 offset: 0 snap -1 2024-09-13T13:51:18.219 INFO:tasks.rados.rados.0.smithi159.stdout:1385: write oid 13 current snap is 28 2024-09-13T13:51:18.219 INFO:tasks.rados.rados.0.smithi159.stdout:1378: finishing write tid 1 to smithi15938319-39 2024-09-13T13:51:18.219 INFO:tasks.rados.rados.0.smithi159.stdout:1385: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-13T13:51:18.225 INFO:tasks.rados.rados.0.smithi159.stdout:1385: writing smithi15938319-13 from 601750 to 1137296 tid 1 2024-09-13T13:51:18.231 INFO:tasks.rados.rados.0.smithi159.stdout:1385: writing smithi15938319-13 from 1929910 to 2201750 tid 2 2024-09-13T13:51:18.231 INFO:tasks.rados.rados.0.smithi159.stdout:1386: set_chunk oid 40 target oid 2024-09-13T13:51:18.232 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:18.231+0000 7f1a526ee640 -1 osd.13 100 handle_osd_map injecting map crc failure 2024-09-13T13:51:18.246 INFO:tasks.rados.rados.0.smithi159.stdout:1386: set_chunk oid 40 offset: 1020416 length: 766464 target oid e4635bc15457831b4e846f31a7ab9a8c4d0b110f6f55b3aa852e16c1b87ea5cc offset: 0 snap -1 2024-09-13T13:51:18.247 INFO:tasks.rados.rados.0.smithi159.stdout:1387: write (excl) oid 7 current snap is 28 2024-09-13T13:51:18.247 INFO:tasks.rados.rados.0.smithi159.stdout:1387: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-13T13:51:18.254 INFO:tasks.rados.rados.0.smithi159.stdout:1387: writing smithi15938319-7 from 618557 to 1345705 tid 1 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1387: writing smithi15938319-7 from 2104745 to 2218557 tid 2 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1388: tier_flush oid 41 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1388: tier_flush oid 41 snap -1 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1378: finishing write tid 2 to smithi15938319-39 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1378: finishing write tid 3 to smithi15938319-39 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1378: finishing write tid 4 to smithi15938319-39 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1378: oid 39 updating version 0 to 163 2024-09-13T13:51:18.259 INFO:tasks.rados.rados.0.smithi159.stdout:1378: oid 39 updating version 163 to 165 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:1378: oid 39 version 165 is already newer than 164 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 165 (ObjNum 248 snap 28 seq_num 248) dirty exists 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:1378: left oid 39 (ObjNum 248 snap 28 seq_num 248) 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:1378: done (10 left) 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:1389: write oid 23 current snap is 28 2024-09-13T13:51:18.260 INFO:tasks.rados.rados.0.smithi159.stdout:1389: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-13T13:51:18.264 INFO:tasks.rados.rados.0.smithi159.stdout:1389: writing smithi15938319-23 from 635364 to 1037761 tid 1 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1389: writing smithi15938319-23 from 1763227 to 2235364 tid 2 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1390: read oid 29 snap 20 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1390: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1391: read oid 33 snap -1 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1391: expect deleted 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1392: read oid 28 snap -1 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1392: expect deleted 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1393: read oid 38 snap -1 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1393: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:18.271 INFO:tasks.rados.rados.0.smithi159.stdout:1394: set_chunk oid 5 target oid 2024-09-13T13:51:18.296 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:18.292+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:18.297 INFO:tasks.rados.rados.0.smithi159.stdout:1394: set_chunk oid 5 offset: 2565632 length: 584704 target oid 316fe1df71fc76922f7faef090d733e74d138ef12f93cb16d3ca019aaac23339 offset: 0 snap -1 2024-09-13T13:51:18.297 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:18.297 INFO:tasks.rados.rados.0.smithi159.stdout:1379: expect (ObjNum 191 snap 18 seq_num 191) 2024-09-13T13:51:18.305 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:18.305 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:18.305 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:18.334 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3002441 2024-09-13T13:51:18.393 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:18.395 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:18.394+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:18.420 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:18.421 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:18.421 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:18.450 INFO:tasks.rados.rados.0.smithi159.stdout:1381: finishing write tid 1 to smithi15938319-42 2024-09-13T13:51:18.450 INFO:tasks.rados.rados.0.smithi159.stdout:1381: finishing write tid 2 to smithi15938319-42 2024-09-13T13:51:18.450 INFO:tasks.rados.rados.0.smithi159.stdout:1380: expect (ObjNum 230 snap 25 seq_num 230) 2024-09-13T13:51:18.496 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:18.495+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:18.538 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:200 4.e 4:709ad093:::smithi15938319-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.804322757999998, 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075516929999999999, 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.539 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 200 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732825+0000", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732826+0000", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2300000000000002e-07 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732828+0000", 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.328e-06 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732833+0000", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7269999999999998e-06 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.732838+0000", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2460000000000003e-06 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734037+0000", 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00119889 2024-09-13T13:51:18.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.734078+0000", 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1675000000000003e-05 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740269+0000", 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00619018 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740318+0000", 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9521999999999998e-05 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.740377+0000", 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8801999999999997e-05 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:217 3.1 e41/33 3:90dfe591:::smithi15938319-18 low_tier:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:18.544 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.753155868999997, 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0097209170000000008, 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783992+0000", 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.783993+0000", 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787823+0000", 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038302620000000001 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787880+0000", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6681999999999997e-05 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787888+0000", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6270000000000006e-06 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787940+0000", 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2132000000000002e-05 2024-09-13T13:51:18.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787966+0000", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5741999999999998e-05 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793657+0000", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0056903739999999998 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.793713+0000", 2024-09-13T13:51:18.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6449000000000001e-05 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:223 3.1 e41/33 3:932c05fb:::smithi15938319-24 low_tier:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.749317634999997, 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.009035576, 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.549 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787830+0000", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.787831+0000", 2024-09-13T13:51:18.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.790933+0000", 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031022459999999999 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791063+0000", 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000129374 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791071+0000", 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0020000000000006e-06 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791120+0000", 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9348999999999998e-05 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.791135+0000", 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5108e-05 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796837+0000", 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057013999999999997 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.796866+0000", 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9295999999999999e-05 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.553 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:228 3.1 e41/33 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'5, mlcod=0'0)", 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.739369496999998, 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011341255999999999, 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797778+0000", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.797780+0000", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.283e-06 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800442+0000", 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026622740000000001 2024-09-13T13:51:18.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800472+0000", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9765999999999999e-05 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800477+0000", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2429999999999996e-06 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800515+0000", 2024-09-13T13:51:18.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7750000000000003e-05 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.800533+0000", 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8657000000000001e-05 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809070+0000", 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085367659999999995 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.809120+0000", 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9517e-05 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:220 3.1 e41/33 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'3)", 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.733002549999995, 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015696446999999999, 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804145+0000", 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.804147+0000", 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1629999999999999e-06 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810603+0000", 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0064567410000000002 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810700+0000", 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6526000000000004e-05 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810708+0000", 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9009999999999994e-06 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810749+0000", 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1288999999999997e-05 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810767+0000", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7798000000000001e-05 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819801+0000", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090344989999999997 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.819842+0000", 2024-09-13T13:51:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.053e-05 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:231 3.1 e41/33 3:9f3f9b00:::smithi15938319-32 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.726536250999999, 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012065741, 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810612+0000", 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.810613+0000", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8400000000000002e-07 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813879+0000", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003266658 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813938+0000", 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8579000000000003e-05 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813945+0000", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3540000000000002e-06 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813991+0000", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5203000000000001e-05 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.814007+0000", 2024-09-13T13:51:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6850999999999998e-05 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822644+0000", 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086365629999999999 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822677+0000", 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3549e-05 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:232 3.1 e41/33 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.723263609, 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0088135090000000006, 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813884+0000", 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.813885+0000", 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6899999999999997e-07 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817413+0000", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035275020000000001 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817568+0000", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000155131 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817574+0000", 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4479999999999999e-06 2024-09-13T13:51:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817639+0000", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5221000000000005e-05 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817657+0000", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7450000000000001e-05 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822683+0000", 2024-09-13T13:51:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050262070000000004 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.822698+0000", 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4881e-05 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:235 3.1 e41/33 3:9761ca94:::smithi15938319-36 low_tier:head v 41'9, mlcod=41'4)", 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.719728580999998, 2024-09-13T13:51:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090169459999999996, 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817419+0000", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.817420+0000", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2300000000000001e-07 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821383+0000", 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039627890000000004 2024-09-13T13:51:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821417+0000", 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4496999999999999e-05 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821424+0000", 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0319999999999998e-06 2024-09-13T13:51:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821470+0000", 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5748000000000003e-05 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.821488+0000", 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8207e-05 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826398+0000", 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004909409 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.826436+0000", 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8640999999999998e-05 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:242 3.1 e41/33 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'11, mlcod=41'8)", 2024-09-13T13:51:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.705663786000002, 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083574370000000005, 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831484+0000", 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.831485+0000", 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2099999999999995e-07 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834193+0000", 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027077939999999999 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834220+0000", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7018000000000001e-05 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834229+0000", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7849999999999997e-06 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834275+0000", 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5961000000000003e-05 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.834296+0000", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1401999999999999e-05 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839806+0000", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055102500000000004 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.839842+0000", 2024-09-13T13:51:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5305999999999997e-05 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:258 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.638964647999998, 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011133136, 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 258 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898183+0000", 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898185+0000", 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7400000000000001e-06 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898191+0000", 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7420000000000003e-06 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898205+0000", 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4628e-05 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898213+0000", 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5020000000000003e-06 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898410+0000", 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000197235 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898490+0000", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9856000000000003e-05 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.904838+0000", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063482139999999996 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.905236+0000", 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039744999999999998 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909250+0000", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0040139110000000002 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909281+0000", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1078999999999998e-05 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909316+0000", 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5778999999999997e-05 2024-09-13T13:51:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:248 3.1 e41/33 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'13, mlcod=41'11)", 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.629949851999996, 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012034909999999999, 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907198+0000", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.907199+0000", 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4499999999999996e-07 2024-09-13T13:51:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911325+0000", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041262089999999996 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911350+0000", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4960000000000002e-05 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911356+0000", 2024-09-13T13:51:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4770000000000002e-06 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911406+0000", 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0633000000000001e-05 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911422+0000", 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5937999999999999e-05 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919153+0000", 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077307970000000002 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.919233+0000", 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0050999999999996e-05 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:299 4.e 4:709ad093:::smithi15938319-50:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.767709257999996, 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0089831609999999999, 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 299 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769439+0000", 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769441+0000", 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5830000000000001e-06 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769453+0000", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1756000000000001e-05 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769498+0000", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4468999999999997e-05 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769510+0000", 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2969000000000001e-05 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7052000000000005e-05 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769700+0000", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2683999999999996e-05 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777412+0000", 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0077117100000000001 2024-09-13T13:51:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.777907+0000", 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00049540900000000004 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778307+0000", 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00039943500000000001 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778372+0000", 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5153000000000005e-05 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778422+0000", 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9941000000000002e-05 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:358 4.e 4:70cd1372:::smithi15938319-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 64.462684948000003, 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086044410000000009, 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 358 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074463+0000", 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074464+0000", 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3179999999999999e-06 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074474+0000", 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5670000000000001e-06 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074488+0000", 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3889e-05 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074497+0000", 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5829999999999996e-06 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074595+0000", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7899999999999994e-05 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.074675+0000", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9849000000000001e-05 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.077944+0000", 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003269044 2024-09-13T13:51:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082062+0000", 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041176290000000003 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.082985+0000", 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00092279199999999997 2024-09-13T13:51:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083021+0000", 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6386e-05 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.083067+0000", 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6483999999999997e-05 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:807 4.e 4:717ade62:::smithi15938319-24:head [write 541192~628415,stat] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 60.356330878000001, 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010708927, 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 807 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180817+0000", 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.180820+0000", 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7879999999999998e-06 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181481+0000", 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00066089999999999996 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181501+0000", 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0503e-05 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181511+0000", 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9329999999999996e-06 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181582+0000", 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0916999999999999e-05 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.181588+0000", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1160000000000004e-06 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185876+0000", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042876900000000003 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.185941+0000", 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5168999999999999e-05 2024-09-13T13:51:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.186064+0000", 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012253599999999999 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191138+0000", 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0050743660000000003 2024-09-13T13:51:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191450+0000", 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00031216700000000002 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191481+0000", 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0954000000000002e-05 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:18.191526+0000", 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4888000000000001e-05 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1550 4.e 4:721f6cf4:::smithi15938319-18:6 [read 1158216~2203151,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.455145852999998, 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090284739999999999, 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1550 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082002+0000", 2024-09-13T13:51:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082003+0000", 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3540000000000001e-06 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082013+0000", 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1330000000000005e-06 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082033+0000", 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9996e-05 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.082048+0000", 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5593000000000001e-05 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088784+0000", 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0067358349999999999 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.088911+0000", 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126907 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.091031+0000", 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021196560000000001 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1552 3.1 e59/33 3:bd95eec2:::8d86d8fa5fbd187d6a1fe9b00c3263164764983256dba23b50442c7883957338:head v 59'84, mlcod=59'82)", 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.444367034000003, 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0081256560000000002, 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092781+0000", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.092783+0000", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6929999999999999e-06 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096023+0000", 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032406370000000002 2024-09-13T13:51:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096060+0000", 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7160000000000003e-05 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096070+0000", 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5540000000000003e-06 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096139+0000", 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9385000000000001e-05 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.096161+0000", 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1483000000000001e-05 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100843+0000", 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046821320000000003 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.100907+0000", 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3612e-05 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1583 4.e 4:721f6cf4:::smithi15938319-18:head [read 0~3361367,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.078124076999998, 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00774357, 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1583 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459024+0000", 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.29e-07 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459029+0000", 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7890000000000002e-06 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459033+0000", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7409999999999998e-06 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.459036+0000", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2279999999999998e-06 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464333+0000", 2024-09-13T13:51:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052973910000000003 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.464381+0000", 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7741000000000003e-05 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.466767+0000", 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0023864509999999999 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1598 4.e 4:717ade62:::smithi15938319-24:head [write 699191~532079] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.021299519000003, 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0075604720000000004, 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1598 2024-09-13T13:51:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515848+0000", 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.515850+0000", 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.449e-06 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516458+0000", 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060838899999999998 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516474+0000", 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5293000000000001e-05 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516481+0000", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9039999999999992e-06 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516553+0000", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1365999999999994e-05 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516663+0000", 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000109738 2024-09-13T13:51:18.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.516884+0000", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022128499999999999 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.519137+0000", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0022529389999999998 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523336+0000", 2024-09-13T13:51:18.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041994830000000004 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523368+0000", 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1813000000000002e-05 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:35.523409+0000", 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0812999999999997e-05 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2147 4.0 e74/34 4:04ca156e:::smithi15938319-49:head v 74'29, mlcod=74'26)", 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 27.731352060999999, 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.79346728, 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805796+0000", 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6899999999999997e-07 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:50.805802+0000", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7080000000000002e-06 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598451+0000", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.79264855599999995 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598460+0000", 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1279999999999997e-06 2024-09-13T13:51:18.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598527+0000", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7605000000000001e-05 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.598548+0000", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0653999999999999e-05 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.599224+0000", 2024-09-13T13:51:18.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00067555200000000005 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.599263+0000", 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9508000000000002e-05 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2313 4.e 4:717ade62:::smithi15938319-24:head [write 726102~788786,stat] snapc 13=[13,12,f,b,8] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 22.767692146000002, 2024-09-13T13:51:18.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011702211000000001, 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2313 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:18.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769456+0000", 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.769457+0000", 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:18.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770373+0000", 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00091632300000000003 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770390+0000", 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6316000000000001e-05 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770408+0000", 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7907999999999999e-05 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770489+0000", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1071999999999999e-05 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.770495+0000", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3280000000000002e-06 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773592+0000", 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003096723 2024-09-13T13:51:18.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773668+0000", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6488000000000004e-05 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.773805+0000", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000136291 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.776916+0000", 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003111688 2024-09-13T13:51:18.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781046+0000", 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041299240000000001 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781111+0000", 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4973000000000003e-05 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:18.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.781158+0000", 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6839999999999999e-05 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:18.644 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:18.646 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:18.644+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:18.731 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:18.729+0000 7f7c73c21640 -1 osd.14 101 handle_osd_map injecting map crc failure 2024-09-13T13:51:18.746 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:18.745+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:18.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:18.755 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:18.755 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:18.755 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:1381: finishing write tid 3 to smithi15938319-42 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:1381: finishing write tid 4 to smithi15938319-42 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:1381: oid 42 updating version 0 to 105 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:1381: oid 42 updating version 105 to 107 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:1381: oid 42 version 107 is already newer than 106 2024-09-13T13:51:18.763 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 107 (ObjNum 249 snap 28 seq_num 249) dirty exists 2024-09-13T13:51:18.764 INFO:tasks.rados.rados.0.smithi159.stdout:1381: left oid 42 (ObjNum 249 snap 28 seq_num 249) 2024-09-13T13:51:18.764 INFO:tasks.rados.rados.0.smithi159.stdout:1384: finishing set_chunk to oid 6 2024-09-13T13:51:18.764 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 29 (ObjNum 182 snap 17 seq_num 182) dirty exists 2024-09-13T13:51:18.764 INFO:tasks.rados.rados.0.smithi159.stdout:1382: expect (ObjNum 224 snap 24 seq_num 224) 2024-09-13T13:51:18.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:18.850 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:18.850+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:18.889 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:18.889 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:18.911 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:18.912 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:18.915 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:18.916 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:18.916 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:18.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:18.933 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:18.955 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:18.955 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:18.956 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:18.955+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:18.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:18.997 INFO:tasks.rados.rados.0.smithi159.stdout:1385: finishing write tid 1 to smithi15938319-13 2024-09-13T13:51:18.997 INFO:tasks.rados.rados.0.smithi159.stdout:1383: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:19.008 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2063635 2024-09-13T13:51:19.018 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:19.030 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:19.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:19.049 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:19.049+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:19.055 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 792552 2024-09-13T13:51:19.057 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:19.057 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:19.069 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.901805236000001, 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.071 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.072 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:51:19.073 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:51:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:51:19.078 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.079 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.901779242999993, 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:51:19.080 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:51:19.083 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.084 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:51:19.085 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:19 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.896833721999997, 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096180398, 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.086 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170582+0000", 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.087 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170583+0000", 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.170585+0000", 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1090000000000001e-06 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266590+0000", 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.096005323000000004 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.088 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266598+0000", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8110000000000003e-06 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266650+0000", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2481000000000001e-05 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266658+0000", 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.126e-06 2024-09-13T13:51:19.089 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266763+0000", 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000105209 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.796470420999995, 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.090 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.091 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.092 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:51:19.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.094 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:111 4.2 e41/34 4:4d382039:::smithi15938319-28:head v 41'9, mlcod=41'5)", 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.623430522000007, 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.06900017, 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443986+0000", 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1600000000000001e-07 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.096 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.443991+0000", 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5630000000000002e-06 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512563+0000", 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.068572151999999997 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.097 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512567+0000", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7119999999999999e-06 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512607+0000", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9249000000000003e-05 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512623+0000", 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5866999999999999e-05 2024-09-13T13:51:19.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512961+0000", 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00033803199999999999 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.512986+0000", 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5079000000000002e-05 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.099 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.500435244000002, 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:51:19.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.101 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.102 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:51:19.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.105 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.500420743999996, 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:51:19.106 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.107 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:51:19.108 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:51:19.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:51:19.111 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.498167304000006, 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:51:19.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:51:19.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:51:19.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.375744015999999, 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.116 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:19.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:51:19.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.375713196999996, 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.122 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:19.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:51:19.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.126 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:19.127 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.373555089000007, 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:51:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.164020891000007, 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:19.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.133 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:51:19.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.298043359000005, 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.136 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:51:19.138 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:19.139 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:51:19.141 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 65.284367645000003, 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:51:19.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.268466061000002, 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:51:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:51:19.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:51:19.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.154 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.260174566, 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.155 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:51:19.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:51:19.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.256147644999999, 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.163 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:51:19.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:51:19.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.170 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.255822688999999, 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:51:19.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:51:19.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:51:19.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.178 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 17.255791164000001, 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:51:19.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:51:19.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 14.228101163, 2024-09-13T13:51:19.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:51:19.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:19.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:19.189 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:19.190 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:19.096+0000 7f8f5ac65640 -1 osd.6 101 handle_osd_map injecting map crc failure 2024-09-13T13:51:19.191 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:19.189+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:19.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:19.192 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:19.193 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:19.193 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:19.194 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:19.194 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:19.194 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:19.195 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:19.195 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:19.195 INFO:tasks.rados.rados.0.smithi159.stdout:1385: finishing write tid 2 to smithi15938319-13 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1385: finishing write tid 3 to smithi15938319-13 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1386: finishing set_chunk to oid 40 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 121 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1387: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1379: done (15 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1380: done (14 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1381: done (13 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1382: done (12 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1383: done (11 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1384: done (10 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1386: done (9 left) 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1395: set_chunk oid 16 target oid 2024-09-13T13:51:19.196 INFO:tasks.rados.rados.0.smithi159.stdout:1395: set_chunk oid 16 offset: 2460672 length: 369152 target oid e062e0024592e94ebcabebfed421cfa3cff8bd631d11371781756e776eb2cbb9 offset: 0 snap 17 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1396: write (excl) oid 6 current snap is 28 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1388: got (16) Device or resource busy 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 41 v 170 (ObjNum 119 snap 8 seq_num 119) unflushed 28 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1387: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1387: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1396: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1396: writing smithi15938319-6 from 652171 to 1129817 tid 1 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1396: writing smithi15938319-6 from 1821709 to 2252171 tid 2 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1388: done (10 left) 2024-09-13T13:51:19.197 INFO:tasks.rados.rados.0.smithi159.stdout:1397: read oid 43 snap -1 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1397: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1398: snap_create 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1387: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1387: oid 7 updating version 0 to 202 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1387: oid 7 updating version 202 to 203 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1387: oid 7 version 203 is already newer than 201 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 203 (ObjNum 251 snap 28 seq_num 251) dirty exists 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1387: left oid 7 (ObjNum 251 snap 28 seq_num 251) 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1389: finishing write tid 1 to smithi15938319-23 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1389: finishing write tid 2 to smithi15938319-23 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1389: finishing write tid 3 to smithi15938319-23 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1389: finishing write tid 4 to smithi15938319-23 2024-09-13T13:51:19.198 INFO:tasks.rados.rados.0.smithi159.stdout:1389: oid 23 updating version 0 to 173 2024-09-13T13:51:19.199 INFO:tasks.rados.rados.0.smithi159.stdout:1389: oid 23 updating version 173 to 175 2024-09-13T13:51:19.199 INFO:tasks.rados.rados.0.smithi159.stdout:1389: oid 23 version 175 is already newer than 174 2024-09-13T13:51:19.199 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 175 (ObjNum 252 snap 28 seq_num 252) dirty exists 2024-09-13T13:51:19.199 INFO:tasks.rados.rados.0.smithi159.stdout:1389: left oid 23 (ObjNum 252 snap 28 seq_num 252) 2024-09-13T13:51:19.199 INFO:tasks.rados.rados.0.smithi159.stdout:1390: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:19.199 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:19.199 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:19.200 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:19.200 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:19.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:19.213 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:19.212+0000 7f1f37764640 -1 osd.4 101 handle_osd_map injecting map crc failure 2024-09-13T13:51:19.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:19.213+0000 7fbf93fff640 0 -- 172.21.15.27:0/2071866759 >> v2:172.21.15.122:6808/1255553857 conn(0x7fbf94221a50 msgr2=0x7fbf94223ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:19.235 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:19.248 INFO:tasks.rados.rados.0.smithi159.stdout:1393: expect (ObjNum 212 snap 22 seq_num 212) 2024-09-13T13:51:19.248 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:19.256 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:19.308 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:19.308+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:19.316 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.317 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:19.392 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:19.391+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:19.470 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.471 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:19.490 INFO:tasks.rados.rados.0.smithi159.stdout:1394: finishing set_chunk to oid 5 2024-09-13T13:51:19.490 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 86 (ObjNum 194 snap 18 seq_num 194) dirty exists 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1385: finishing write tid 4 to smithi15938319-13 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1385: oid 13 updating version 0 to 107 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1385: oid 13 updating version 107 to 108 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1385: oid 13 version 108 is already newer than 106 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 108 (ObjNum 250 snap 28 seq_num 250) dirty exists 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1385: left oid 13 (ObjNum 250 snap 28 seq_num 250) 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1395: finishing set_chunk to oid 16 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: finishing write tid 1 to smithi15938319-6 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: finishing write tid 2 to smithi15938319-6 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: finishing write tid 3 to smithi15938319-6 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: finishing write tid 4 to smithi15938319-6 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: oid 6 updating version 0 to 38 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: oid 6 version 38 is already newer than 36 2024-09-13T13:51:19.491 INFO:tasks.rados.rados.0.smithi159.stdout:1396: oid 6 version 38 is already newer than 37 2024-09-13T13:51:19.492 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 38 (ObjNum 253 snap 28 seq_num 253) dirty exists 2024-09-13T13:51:19.492 INFO:tasks.rados.rados.0.smithi159.stdout:1396: left oid 6 (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:51:19.492 INFO:tasks.rados.rados.0.smithi159.stdout:1397: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:19.493 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:19.492+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:19.555 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:19.586 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.586 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.594 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:19.592+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:19.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:19.695 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:19.694+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:19.796 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:19.796+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:19.821 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.822 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:19.886 INFO:tasks.rados.rados.0.smithi159.stdout:1385: done (11 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1387: done (10 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1389: done (9 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1390: done (8 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1391: done (7 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1392: done (6 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1393: done (5 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1394: done (4 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1395: done (3 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1396: done (2 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1397: done (1 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1398: done (0 left) 2024-09-13T13:51:19.887 INFO:tasks.rados.rados.0.smithi159.stdout:1399: snap_create 2024-09-13T13:51:19.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:51:19.896 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:19.895+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:19.933 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:19.934 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:19.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:19.998 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:19.997+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:20.046 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.046 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.055 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:20.099 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:20.098+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:20.156 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.156 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:20.199 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:20.198+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:20.267 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.268 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:20.300 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:20.299+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:20.378 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.378 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.386 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:20.430 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:20.429+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:20.489 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.489 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.497 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:20.502 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:20.501+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:20.602 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:20.601+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:20.703 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:20.703+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:20.793 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.793 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.802 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:20.803 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:20.803+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:20.825 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:20.823+0000 7f7c71c1d640 -1 osd.14 102 handle_osd_map injecting map crc failure 2024-09-13T13:51:20.893 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:51:20.894 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:20.893+0000 7faeac214640 -1 osd.8 102 handle_osd_map injecting map crc failure 2024-09-13T13:51:20.898 INFO:tasks.rados.rados.0.smithi159.stdout:1399: done (0 left) 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1400: read oid 37 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1400: expect deleted 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1401: tier_flush oid 13 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1401: tier_flush oid 13 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1402: read oid 15 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1402: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1403: read oid 5 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1403: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1404: read oid 43 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1404: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1405: read oid 49 snap -1 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1405: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:20.899 INFO:tasks.rados.rados.0.smithi159.stdout:1406: read oid 22 snap -1 2024-09-13T13:51:20.900 INFO:tasks.rados.rados.0.smithi159.stdout:1406: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:51:20.900 INFO:tasks.rados.rados.0.smithi159.stdout:1407: rollback oid 9 current snap is 30 2024-09-13T13:51:20.900 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 9 to 29 2024-09-13T13:51:20.900 INFO:tasks.rados.rados.0.smithi159.stdout:1408: write (excl) oid 3 current snap is 30 2024-09-13T13:51:20.900 INFO:tasks.rados.rados.0.smithi159.stdout:1408: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-13T13:51:20.904 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:20.902+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:20.905 INFO:tasks.rados.rados.0.smithi159.stdout:1408: writing smithi15938319-3 from 668978 to 1221873 tid 1 2024-09-13T13:51:20.906 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:20.906 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:20.911 INFO:tasks.rados.rados.0.smithi159.stdout:1408: writing smithi15938319-3 from 1763838 to 2268978 tid 2 2024-09-13T13:51:20.911 INFO:tasks.rados.rados.0.smithi159.stdout:1409: write oid 41 current snap is 30 2024-09-13T13:51:20.911 INFO:tasks.rados.rados.0.smithi159.stdout:1409: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-13T13:51:20.918 INFO:tasks.rados.rados.0.smithi159.stdout:1409: writing smithi15938319-41 from 685785 to 1313929 tid 1 2024-09-13T13:51:20.919 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:20.923 INFO:tasks.rados.rados.0.smithi159.stdout:1409: writing smithi15938319-41 from 1822320 to 2285785 tid 2 2024-09-13T13:51:20.923 INFO:tasks.rados.rados.0.smithi159.stdout:1410: write oid 42 current snap is 30 2024-09-13T13:51:20.923 INFO:tasks.rados.rados.0.smithi159.stdout:1410: seq_num 256 ranges {702592=703393,1880802=421790} 2024-09-13T13:51:20.931 INFO:tasks.rados.rados.0.smithi159.stdout:1410: writing smithi15938319-42 from 702592 to 1405985 tid 1 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:1410: writing smithi15938319-42 from 1880802 to 2302592 tid 2 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:1400: done (10 left) 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:1411: tier_evict oid 34 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:1411: tier_evict oid 34 snap -1 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:1401: got (16) Device or resource busy 2024-09-13T13:51:20.936 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 13 v 108 (ObjNum 250 snap 28 seq_num 250) unflushed 28 2024-09-13T13:51:20.937 INFO:tasks.rados.rados.0.smithi159.stdout:1401: done (10 left) 2024-09-13T13:51:20.937 INFO:tasks.rados.rados.0.smithi159.stdout:1412: write oid 8 current snap is 30 2024-09-13T13:51:20.937 INFO:tasks.rados.rados.0.smithi159.stdout:1412: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-13T13:51:20.943 INFO:tasks.rados.rados.0.smithi159.stdout:1412: writing smithi15938319-8 from 719399 to 1498041 tid 1 2024-09-13T13:51:20.948 INFO:tasks.rados.rados.0.smithi159.stdout:1412: writing smithi15938319-8 from 1939284 to 2319399 tid 2 2024-09-13T13:51:20.948 INFO:tasks.rados.rados.0.smithi159.stdout:1413: set_chunk oid 18 target oid 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:1413: set_chunk oid 18 offset: 285696 length: 1544704 target oid f0ac9db464fe05be30c599bc52a69eeeb746e8679eaf804acdaf7540e06a450c offset: 0 snap -1 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:1414: rollback oid 40 current snap is 30 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 40 to 16 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:1407: finishing rollback tid 0 to smithi15938319-9 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:1407: finishing rollback tid 1 to smithi15938319-9 2024-09-13T13:51:20.968 INFO:tasks.rados.rados.0.smithi159.stdout:1407: finishing rollback tid 2 to smithi15938319-9 2024-09-13T13:51:20.969 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 200 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:51:20.969 INFO:tasks.rados.rados.0.smithi159.stdout:1403: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:21.005 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:21.003+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:21.024 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:21.024 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:21.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:21.106 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:21.106+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:21.122 INFO:tasks.rados.rados.0.smithi159.stdout:1406: expect (ObjNum 153 snap 12 seq_num 153) 2024-09-13T13:51:21.123 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:21.142 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:21.142 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:21.151 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:21.180 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 855578 2024-09-13T13:51:21.198 INFO:tasks.rados.rados.0.smithi159.stdout:1404: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:21.208 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:21.206+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:21.254 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:21.254 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:21.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:21.268 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:21.308 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:21.307+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:21.313 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1284093 2024-09-13T13:51:21.364 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:21.364 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:21.365 INFO:tasks.rados.rados.0.smithi159.stdout:1405: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:21.369 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1340993 2024-09-13T13:51:21.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:21.409 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:21.408+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:21.411 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1245379 2024-09-13T13:51:21.413 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:21.413 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:21.436 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:21.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:21.458 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:21.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:21.479 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:21.479 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:21.485 INFO:tasks.rados.rados.0.smithi159.stdout:1408: finishing write tid 1 to smithi15938319-3 2024-09-13T13:51:21.486 INFO:tasks.rados.rados.0.smithi159.stdout:1402: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:21.489 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 664636 2024-09-13T13:51:21.499 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:21.499 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:21.509 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:21.509+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:21.517 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:21.517 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:21.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:21.526 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:21.526 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:21.533 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:21.547 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:21.547 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:21.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:21.585 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 209844 2024-09-13T13:51:21.589 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:21.589 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:21.610 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:21.611 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:21.611 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:21.610+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:21.629 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:21.629 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:21.629 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:21.629 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:21.629 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:21.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:21.631 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: finishing write tid 2 to smithi15938319-3 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: finishing write tid 3 to smithi15938319-3 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: finishing write tid 4 to smithi15938319-3 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: oid 3 updating version 0 to 205 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: oid 3 updating version 205 to 207 2024-09-13T13:51:21.638 INFO:tasks.rados.rados.0.smithi159.stdout:1408: oid 3 version 207 is already newer than 206 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 207 (ObjNum 254 snap 30 seq_num 254) dirty exists 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1408: left oid 3 (ObjNum 254 snap 30 seq_num 254) 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: finishing write tid 1 to smithi15938319-41 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: finishing write tid 2 to smithi15938319-41 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: finishing write tid 3 to smithi15938319-41 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: finishing write tid 4 to smithi15938319-41 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: oid 41 updating version 0 to 174 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: oid 41 version 174 is already newer than 172 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: oid 41 version 174 is already newer than 173 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 174 (ObjNum 255 snap 30 seq_num 255) dirty exists 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1409: left oid 41 (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1410: finishing write tid 1 to smithi15938319-42 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1410: finishing write tid 2 to smithi15938319-42 2024-09-13T13:51:21.639 INFO:tasks.rados.rados.0.smithi159.stdout:1410: finishing write tid 3 to smithi15938319-42 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1402: done (12 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1403: done (11 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1404: done (10 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1405: done (9 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1406: done (8 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1407: done (7 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1408: done (6 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1409: done (5 left) 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1415: read oid 46 snap 20 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1415: expect deleted 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1416: rollback oid 31 current snap is 30 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 31 to 18 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1410: finishing write tid 4 to smithi15938319-42 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1410: oid 42 updating version 0 to 111 2024-09-13T13:51:21.640 INFO:tasks.rados.rados.0.smithi159.stdout:1410: oid 42 version 111 is already newer than 109 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1410: oid 42 version 111 is already newer than 110 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 111 (ObjNum 256 snap 30 seq_num 256) dirty exists 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1410: left oid 42 (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: finishing write tid 1 to smithi15938319-8 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: finishing write tid 2 to smithi15938319-8 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: finishing write tid 3 to smithi15938319-8 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: finishing write tid 4 to smithi15938319-8 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: oid 8 updating version 0 to 114 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: oid 8 version 114 is already newer than 113 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: oid 8 updating version 114 to 115 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 115 (ObjNum 257 snap 30 seq_num 257) dirty exists 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1412: left oid 8 (ObjNum 257 snap 30 seq_num 257) 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:1413: finishing set_chunk to oid 18 2024-09-13T13:51:21.641 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 182 (ObjNum 242 snap 28 seq_num 242) dirty exists 2024-09-13T13:51:21.642 INFO:tasks.rados.rados.0.smithi159.stdout:1414: finishing rollback tid 0 to smithi15938319-40 2024-09-13T13:51:21.642 INFO:tasks.rados.rados.0.smithi159.stdout:1410: done (6 left) 2024-09-13T13:51:21.642 INFO:tasks.rados.rados.0.smithi159.stdout:1412: done (5 left) 2024-09-13T13:51:21.642 INFO:tasks.rados.rados.0.smithi159.stdout:1413: done (4 left) 2024-09-13T13:51:21.642 INFO:tasks.rados.rados.0.smithi159.stdout:1417: snap_remove snap 29 2024-09-13T13:51:21.643 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:21.652 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:21.673 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:21.673 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:21.693 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:21.694 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:21.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:21.693+0000 7f61f2ffd640 0 -- 172.21.15.27:0/196415307 >> v2:172.21.15.122:6808/1255553857 conn(0x7f61f421fad0 msgr2=0x7f61f4221f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:51:21.711 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:21.710+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:21.748 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:21.748 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:21.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:21.748 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:21.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.204647237000003, 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.749 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:51:21.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:51:21.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:21.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.754 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 91.150321348000006, 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:21.755 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:51:21.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 91.136942958000006, 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.761 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:51:21.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:21.765 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 63.323610021, 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:51:21.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:51:21.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.518198013999999, 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.774 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:51:21.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.779 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.518160842, 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.780 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:51:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:51:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:51:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:51:21.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.518140113999998, 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:21.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:51:21.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.518043812999998, 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.792 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:51:21.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:51:21.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.518021169000001, 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:51:21.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.517995139, 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:51:21.804 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:51:21.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:51:21.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.810 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.378494271999998, 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:51:21.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.378387324999998, 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.817 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:51:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.378358065999997, 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:51:21.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:51:21.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:51:21.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:51:21.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:21.830 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.371993662999998, 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:21.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:51:21.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:51:21.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.367033468999999, 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.839 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:21.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:51:21.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:51:21.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:21.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 46.425043752000001, 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.847 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:51:21.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:51:21.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:51:21.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 46.425004614000002, 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:51:21.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:51:21.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.883574617000001, 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:51:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:51:21.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.883511671000001, 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:51:21.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:21.867 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 19.883477988999999, 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:21.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:51:21.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:21.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:51:21.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:21.872 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:21.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:21.874 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:21.872+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:21.902 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:21.902+0000 7ffaf0cec640 -1 osd.0 104 handle_osd_map injecting map crc failure 2024-09-13T13:51:21.904 INFO:tasks.rados.rados.0.smithi159.stdout:1417: done (4 left) 2024-09-13T13:51:21.904 INFO:tasks.rados.rados.0.smithi159.stdout:1418: tier_promote oid 5 2024-09-13T13:51:21.904 INFO:tasks.rados.rados.0.smithi159.stdout:1414: finishing rollback tid 1 to smithi15938319-40 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1414: finishing rollback tid 2 to smithi15938319-40 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 170 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1411: got (22) Invalid argument 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1416: finishing rollback tid 0 to smithi15938319-31 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1416: finishing rollback tid 1 to smithi15938319-31 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1411: done (4 left) 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1414: done (3 left) 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1415: done (2 left) 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1419: read oid 18 snap 20 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1419: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1420: write oid 26 current snap is 30 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:1416: finishing rollback tid 2 to smithi15938319-31 2024-09-13T13:51:21.905 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 174 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:51:21.906 INFO:tasks.rados.rados.0.smithi159.stdout:1420: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-13T13:51:21.906 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:21.903+0000 7fe0f9e03640 -1 osd.2 104 handle_osd_map injecting map crc failure 2024-09-13T13:51:21.906 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:21.904+0000 7f1f37764640 -1 osd.4 104 handle_osd_map injecting map crc failure 2024-09-13T13:51:21.910 INFO:tasks.rados.rados.0.smithi159.stdout:1420: writing smithi15938319-26 from 736206 to 1190097 tid 1 2024-09-13T13:51:21.917 INFO:tasks.rados.rados.0.smithi159.stdout:1420: writing smithi15938319-26 from 1881413 to 2336206 tid 2 2024-09-13T13:51:21.917 INFO:tasks.rados.rados.0.smithi159.stdout:1416: done (3 left) 2024-09-13T13:51:21.917 INFO:tasks.rados.rados.0.smithi159.stdout:1421: write oid 16 current snap is 30 2024-09-13T13:51:21.917 INFO:tasks.rados.rados.0.smithi159.stdout:1421: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-13T13:51:21.919 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:21.919 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:21.924 INFO:tasks.rados.rados.0.smithi159.stdout:1421: writing smithi15938319-16 from 753013 to 1398506 tid 1 2024-09-13T13:51:21.930 INFO:tasks.rados.rados.0.smithi159.stdout:1421: writing smithi15938319-16 from 2056248 to 2353013 tid 2 2024-09-13T13:51:21.930 INFO:tasks.rados.rados.0.smithi159.stdout:1422: tier_flush oid 50 2024-09-13T13:51:21.930 INFO:tasks.rados.rados.0.smithi159.stdout:1422: tier_flush oid 50 snap -1 2024-09-13T13:51:21.930 INFO:tasks.rados.rados.0.smithi159.stdout:1419: expect (ObjNum 201 snap 19 seq_num 201) 2024-09-13T13:51:21.943 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:21.943 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:21.977 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:21.975+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:21.983 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:21.983 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:21.992 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:22.019 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 612330 2024-09-13T13:51:22.078 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:22.076+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:22.108 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1434093 2024-09-13T13:51:22.134 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:22.134 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:22.134 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:22.134 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:22.134 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:22.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1418: got (0) Success 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 86 (ObjNum 194 snap 18 seq_num 194) dirty exists 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: finishing write tid 1 to smithi15938319-26 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: finishing write tid 2 to smithi15938319-26 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: finishing write tid 3 to smithi15938319-26 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: finishing write tid 4 to smithi15938319-26 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: oid 26 updating version 0 to 100 2024-09-13T13:51:22.165 INFO:tasks.rados.rados.0.smithi159.stdout:1420: oid 26 version 100 is already newer than 98 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1420: oid 26 version 100 is already newer than 99 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 100 (ObjNum 258 snap 30 seq_num 258) dirty exists 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1420: left oid 26 (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1421: finishing write tid 1 to smithi15938319-16 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1418: done (4 left) 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1419: done (3 left) 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1420: done (2 left) 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1423: rollback oid 28 current snap is 30 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 28 to 26 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1422: got (16) Device or resource busy 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 50 v 0 (ObjNum 33 snap 0 seq_num 1540267136) unflushed 24 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1421: finishing write tid 2 to smithi15938319-16 2024-09-13T13:51:22.166 INFO:tasks.rados.rados.0.smithi159.stdout:1421: finishing write tid 3 to smithi15938319-16 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: finishing write tid 4 to smithi15938319-16 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: oid 16 updating version 0 to 232 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: oid 16 version 232 is already newer than 230 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: oid 16 version 232 is already newer than 231 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 232 (ObjNum 259 snap 30 seq_num 259) dirty exists 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: left oid 16 (ObjNum 259 snap 30 seq_num 259) 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1421: done (2 left) 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1422: done (1 left) 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1424: rollback oid 25 current snap is 30 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 25 to 17 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1425: write (excl) oid 19 current snap is 30 2024-09-13T13:51:22.167 INFO:tasks.rados.rados.0.smithi159.stdout:1425: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-13T13:51:22.172 INFO:tasks.rados.rados.0.smithi159.stdout:1425: writing smithi15938319-19 from 769820 to 1490562 tid 1 2024-09-13T13:51:22.182 INFO:tasks.rados.rados.0.smithi159.stdout:1425: writing smithi15938319-19 from 2114730 to 2369820 tid 2 2024-09-13T13:51:22.280 INFO:tasks.rados.rados.0.smithi159.stdout:1426: read oid 2 snap -1 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1426: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1427: read oid 42 snap -1 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1427: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1428: tier_evict oid 11 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1428: tier_evict oid 11 snap -1 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1423: finishing rollback tid 1 to smithi15938319-28 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 149 (ObjNum 1540216784 snap 22089 seq_num 304) dirty dne 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1424: finishing rollback tid 0 to smithi15938319-25 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1423: done (5 left) 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:1429: rollback oid 30 current snap is 30 2024-09-13T13:51:22.281 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 30 to 21 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1424: finishing rollback tid 1 to smithi15938319-25 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1424: finishing rollback tid 2 to smithi15938319-25 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 126 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1424: done (5 left) 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1430: read oid 33 snap -1 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1430: expect deleted 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1431: tier_promote oid 17 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1432: read oid 36 snap -1 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1432: expect deleted 2024-09-13T13:51:22.282 INFO:tasks.rados.rados.0.smithi159.stdout:1433: set_chunk oid 48 target oid 2024-09-13T13:51:22.283 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:51:22.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1433: set_chunk oid 48 offset: 636928 length: 1082368 target oid ed8f65fa18ec0093ab33db5b89efd8d2a97a23a7caca443bc57ffdb7b229ba51 offset: 0 snap -1 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1434: write (excl) oid 8 current snap is 30 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1434: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1434: writing smithi15938319-8 from 786627 to 1582618 tid 1 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1434: writing smithi15938319-8 from 2173212 to 2386627 tid 2 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1435: set_chunk oid 16 target oid 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1435: set_chunk oid 16 offset: 163840 length: 318464 target oid 00814125341f622f3543dddffbcb25b2b689a80617cd258544f4181208dda787 offset: 0 snap -1 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1436: write (excl) oid 47 current snap is 30 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1436: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1436: writing smithi15938319-47 from 403434 to 874674 tid 1 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1436: writing smithi15938319-47 from 1315341 to 1716097 tid 2 2024-09-13T13:51:22.286 INFO:tasks.rados.rados.0.smithi159.stdout:1437: write (excl) oid 12 current snap is 30 2024-09-13T13:51:22.287 INFO:tasks.rados.rados.0.smithi159.stdout:1437: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-13T13:51:22.287 INFO:tasks.rados.rados.0.smithi159.stdout:1437: writing smithi15938319-12 from 420241 to 966730 tid 1 2024-09-13T13:51:22.287 INFO:tasks.rados.rados.0.smithi159.stdout:1437: writing smithi15938319-12 from 1373823 to 1918237 tid 2 2024-09-13T13:51:22.287 INFO:tasks.rados.rados.0.smithi159.stdout:1438: snap_remove snap 20 2024-09-13T13:51:22.287 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:22.283+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:22.384 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:22.383+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:22.391 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.649208912999995, 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:51:22.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.643968290999993, 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.643916977000003, 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:51:22.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:22.411 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.643844779999995, 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:22.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:22.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:51:22.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.643805775000004, 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.418 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:51:22.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:51:22.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.424 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.643741363000004, 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:51:22.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:51:22.431 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.618391267000007, 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.432 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:51:22.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:51:22.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.438 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.587830443000001, 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:51:22.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:51:22.444 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.566503855999997, 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.445 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.449 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.488287579000001, 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:51:22.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:51:22.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.455 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144315919, 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.456 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:22.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:51:22.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.462 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144257867, 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:51:22.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:51:22.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.468 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144231908000002, 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:51:22.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:51:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:22.474 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144117491999999, 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:51:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144086600999998, 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.481 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:51:22.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:22.488 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.144050018000002, 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:51:22.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:51:22.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.064255148000001, 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:51:22.495 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:51:22.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 40.689594730000003, 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:51:22.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:22.503 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.566403207, 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:51:22.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.509 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.558382908999999, 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:51:22.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.514 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.514 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.514 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:22.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:22.516 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:22.514+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:22.585 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:51:22.586 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:51:22.597 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T13:51:22.379093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":325496209,"num_objects":203,"num_object_clones":71,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":6,"num_read":3332,"num_read_kb":2337053,"num_write":1815,"num_write_kb":506770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":292907403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2646,"ondisk_log_size":2646,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":955816,"kb_used_data":498472,"kb_used_omap":24,"kb_used_meta":457063,"kb_avail":1508993624,"statfs":{"total":1546188226560,"available":1545209470976,"internally_reserved":0,"allocated":510435328,"data_stored":600300741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":468032593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13152948,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":-1,"num_read":98,"num_read_kb":78403,"num_write":44,"num_write_kb":10881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000859"},"pg_stats":[{"pgid":"4.f","version":"104'170","reported_seq":524,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.972351+0000","last_change":"2024-09-13T13:51:17.277078+0000","last_active":"2024-09-13T13:51:20.972351+0000","last_peered":"2024-09-13T13:51:20.972351+0000","last_clean":"2024-09-13T13:51:20.972351+0000","last_became_active":"2024-09-13T13:50:06.073131+0000","last_became_peered":"2024-09-13T13:50:06.073131+0000","last_unstale":"2024-09-13T13:51:20.972351+0000","last_undegraded":"2024-09-13T13:51:20.972351+0000","last_fullsized":"2024-09-13T13:51:20.972351+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349367,"stat_sum":{"num_bytes":12634648,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":240,"num_read_kb":186240,"num_write":86,"num_write_kb":21450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.e","version":"100'196","reported_seq":584,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.874564+0000","last_change":"2024-09-13T13:51:16.874510+0000","last_active":"2024-09-13T13:51:17.874564+0000","last_peered":"2024-09-13T13:51:17.874564+0000","last_clean":"2024-09-13T13:51:17.874564+0000","last_became_active":"2024-09-13T13:50:06.072867+0000","last_became_peered":"2024-09-13T13:50:06.072867+0000","last_unstale":"2024-09-13T13:51:17.874564+0000","last_undegraded":"2024-09-13T13:51:17.874564+0000","last_fullsized":"2024-09-13T13:51:17.874564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'178","last_scrub_stamp":"2024-09-13T13:51:13.109829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.109829+0000","objects_scrubbed":9,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:18.981431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0017295500000000001,"stat_sum":{"num_bytes":19057900,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":336,"num_read_kb":251766,"num_write":104,"num_write_kb":27450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.d","version":"102'38","reported_seq":341,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900989+0000","last_change":"2024-09-13T13:51:16.876002+0000","last_active":"2024-09-13T13:51:20.900989+0000","last_peered":"2024-09-13T13:51:20.900989+0000","last_clean":"2024-09-13T13:51:20.900989+0000","last_became_active":"2024-09-13T13:50:06.075524+0000","last_became_peered":"2024-09-13T13:50:06.075524+0000","last_unstale":"2024-09-13T13:51:20.900989+0000","last_undegraded":"2024-09-13T13:51:20.900989+0000","last_fullsized":"2024-09-13T13:51:20.900989+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034429399999999998,"stat_sum":{"num_bytes":5276353,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":43420,"num_write":21,"num_write_kb":5134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.c","version":"104'174","reported_seq":588,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.926519+0000","last_change":"2024-09-13T13:51:16.879999+0000","last_active":"2024-09-13T13:51:20.926519+0000","last_peered":"2024-09-13T13:51:20.926519+0000","last_clean":"2024-09-13T13:51:20.926519+0000","last_became_active":"2024-09-13T13:50:28.729220+0000","last_became_peered":"2024-09-13T13:50:28.729220+0000","last_unstale":"2024-09-13T13:51:20.926519+0000","last_undegraded":"2024-09-13T13:51:20.926519+0000","last_fullsized":"2024-09-13T13:51:20.926519+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'162","last_scrub_stamp":"2024-09-13T13:51:12.406192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:12.406192+0000","objects_scrubbed":8,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:23.557831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004311283,"stat_sum":{"num_bytes":19657354,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":255,"num_read_kb":231014,"num_write":97,"num_write_kb":24394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11131276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.b","version":"99'199","reported_seq":516,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876472+0000","last_change":"2024-09-13T13:51:16.875325+0000","last_active":"2024-09-13T13:51:17.876472+0000","last_peered":"2024-09-13T13:51:17.876472+0000","last_clean":"2024-09-13T13:51:17.876472+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:51:17.876472+0000","last_undegraded":"2024-09-13T13:51:17.876472+0000","last_fullsized":"2024-09-13T13:51:17.876472+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:24.347511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031213399999999999,"stat_sum":{"num_bytes":30278287,"num_objects":15,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":163,"num_read_kb":104738,"num_write":122,"num_write_kb":32130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.a","version":"99'137","reported_seq":573,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131764+0000","last_change":"2024-09-13T13:51:17.131913+0000","last_active":"2024-09-13T13:51:17.131764+0000","last_peered":"2024-09-13T13:51:17.131764+0000","last_clean":"2024-09-13T13:51:17.131764+0000","last_became_active":"2024-09-13T13:51:02.821659+0000","last_became_peered":"2024-09-13T13:51:02.821659+0000","last_unstale":"2024-09-13T13:51:17.131764+0000","last_undegraded":"2024-09-13T13:51:17.131764+0000","last_fullsized":"2024-09-13T13:51:17.131764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414707,"stat_sum":{"num_bytes":24443983,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":260,"num_read_kb":253969,"num_write":75,"num_write_kb":23136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":34232380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.9","version":"95'21","reported_seq":296,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.431579+0000","last_change":"2024-09-13T13:51:17.159367+0000","last_active":"2024-09-13T13:51:17.431579+0000","last_peered":"2024-09-13T13:51:17.431579+0000","last_clean":"2024-09-13T13:51:17.431579+0000","last_became_active":"2024-09-13T13:50:12.197810+0000","last_became_peered":"2024-09-13T13:50:12.197810+0000","last_unstale":"2024-09-13T13:51:17.431579+0000","last_undegraded":"2024-09-13T13:51:17.431579+0000","last_fullsized":"2024-09-13T13:51:17.431579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032133200000000002,"stat_sum":{"num_bytes":2571471,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":40643,"num_write":14,"num_write_kb":3605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5],"acting":[15,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.8","version":"101'108","reported_seq":419,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902697+0000","last_change":"2024-09-13T13:51:17.277230+0000","last_active":"2024-09-13T13:51:20.902697+0000","last_peered":"2024-09-13T13:51:20.902697+0000","last_clean":"2024-09-13T13:51:20.902697+0000","last_became_active":"2024-09-13T13:50:06.075916+0000","last_became_peered":"2024-09-13T13:50:06.075916+0000","last_unstale":"2024-09-13T13:51:20.902697+0000","last_undegraded":"2024-09-13T13:51:20.902697+0000","last_fullsized":"2024-09-13T13:51:20.902697+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:23.021997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043106399999999999,"stat_sum":{"num_bytes":11823525,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":107,"num_read_kb":82752,"num_write":55,"num_write_kb":14502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.5","version":"101'175","reported_seq":427,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.901166+0000","last_change":"2024-09-13T13:51:16.883364+0000","last_active":"2024-09-13T13:51:20.901166+0000","last_peered":"2024-09-13T13:51:20.901166+0000","last_clean":"2024-09-13T13:51:20.901166+0000","last_became_active":"2024-09-13T13:51:05.828626+0000","last_became_peered":"2024-09-13T13:51:05.828626+0000","last_unstale":"2024-09-13T13:51:20.901166+0000","last_undegraded":"2024-09-13T13:51:20.901166+0000","last_fullsized":"2024-09-13T13:51:20.901166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'163","last_scrub_stamp":"2024-09-13T13:51:14.489895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.489895+0000","objects_scrubbed":12,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:28.143616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00034909699999999998,"stat_sum":{"num_bytes":21062400,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":171,"num_read_kb":129526,"num_write":94,"num_write_kb":28485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":40416780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900090+0000","last_change":"2024-09-13T13:51:09.467323+0000","last_active":"2024-09-13T13:51:20.900090+0000","last_peered":"2024-09-13T13:51:20.900090+0000","last_clean":"2024-09-13T13:51:20.900090+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:51:20.900090+0000","last_undegraded":"2024-09-13T13:51:20.900090+0000","last_fullsized":"2024-09-13T13:51:20.900090+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:51:09.467249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.467249+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:36.764015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":171,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876473+0000","last_change":"2024-09-13T13:51:13.789246+0000","last_active":"2024-09-13T13:51:17.876473+0000","last_peered":"2024-09-13T13:51:17.876473+0000","last_clean":"2024-09-13T13:51:17.876473+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:51:17.876473+0000","last_undegraded":"2024-09-13T13:51:17.876473+0000","last_fullsized":"2024-09-13T13:51:17.876473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:51:13.789157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.789157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:16.372966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"95'123","reported_seq":410,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876473+0000","last_change":"2024-09-13T13:50:51.191971+0000","last_active":"2024-09-13T13:51:17.876473+0000","last_peered":"2024-09-13T13:51:17.876473+0000","last_clean":"2024-09-13T13:51:17.876473+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:51:17.876473+0000","last_undegraded":"2024-09-13T13:51:17.876473+0000","last_fullsized":"2024-09-13T13:51:17.876473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:19.528621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20001280,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":100,"num_read_kb":4842,"num_write":132,"num_write_kb":52676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"100'96","reported_seq":469,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.664703+0000","last_change":"2024-09-13T13:51:17.277204+0000","last_active":"2024-09-13T13:51:17.664703+0000","last_peered":"2024-09-13T13:51:17.664703+0000","last_clean":"2024-09-13T13:51:17.664703+0000","last_became_active":"2024-09-13T13:51:00.088253+0000","last_became_peered":"2024-09-13T13:51:00.088253+0000","last_unstale":"2024-09-13T13:51:17.664703+0000","last_undegraded":"2024-09-13T13:51:17.664703+0000","last_fullsized":"2024-09-13T13:51:17.664703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:21.234859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043333299999999999,"stat_sum":{"num_bytes":8233881,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":146,"num_read_kb":129809,"num_write":55,"num_write_kb":15323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14889074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131190+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:51:17.131190+0000","last_peered":"2024-09-13T13:51:17.131190+0000","last_clean":"2024-09-13T13:51:17.131190+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:51:17.131190+0000","last_undegraded":"2024-09-13T13:51:17.131190+0000","last_fullsized":"2024-09-13T13:51:17.131190+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"104'132","reported_seq":290,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.967858+0000","last_change":"2024-09-13T13:50:05.123338+0000","last_active":"2024-09-13T13:51:20.967858+0000","last_peered":"2024-09-13T13:51:20.967858+0000","last_clean":"2024-09-13T13:51:20.967858+0000","last_became_active":"2024-09-13T13:50:05.123078+0000","last_became_peered":"2024-09-13T13:50:05.123078+0000","last_unstale":"2024-09-13T13:51:20.967858+0000","last_undegraded":"2024-09-13T13:51:20.967858+0000","last_fullsized":"2024-09-13T13:51:20.967858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:27.240224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633643,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":108,"num_read_kb":8112,"num_write":145,"num_write_kb":53772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"99'122","reported_seq":461,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.875824+0000","last_change":"2024-09-13T13:51:17.274010+0000","last_active":"2024-09-13T13:51:17.875824+0000","last_peered":"2024-09-13T13:51:17.875824+0000","last_clean":"2024-09-13T13:51:17.875824+0000","last_became_active":"2024-09-13T13:50:29.051980+0000","last_became_peered":"2024-09-13T13:50:29.051980+0000","last_unstale":"2024-09-13T13:51:17.875824+0000","last_undegraded":"2024-09-13T13:51:17.875824+0000","last_fullsized":"2024-09-13T13:51:17.875824+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:23.929566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048694199999999997,"stat_sum":{"num_bytes":15717533,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":106,"num_read_kb":57123,"num_write":76,"num_write_kb":20784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9395884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":170,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876472+0000","last_change":"2024-09-13T13:51:11.801324+0000","last_active":"2024-09-13T13:51:17.876472+0000","last_peered":"2024-09-13T13:51:17.876472+0000","last_clean":"2024-09-13T13:51:17.876472+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:51:17.876472+0000","last_undegraded":"2024-09-13T13:51:17.876472+0000","last_fullsized":"2024-09-13T13:51:17.876472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:11.801231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:11.801231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.494865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"98'120","reported_seq":164,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902480+0000","last_change":"2024-09-13T13:50:50.914930+0000","last_active":"2024-09-13T13:51:20.902480+0000","last_peered":"2024-09-13T13:51:20.902480+0000","last_clean":"2024-09-13T13:51:20.902480+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:51:20.902480+0000","last_undegraded":"2024-09-13T13:51:20.902480+0000","last_fullsized":"2024-09-13T13:51:20.902480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18678541,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":96,"num_read_kb":2325,"num_write":127,"num_write_kb":45722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"104'115","reported_seq":130,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.951184+0000","last_change":"2024-09-13T13:51:16.883427+0000","last_active":"2024-09-13T13:51:20.951184+0000","last_peered":"2024-09-13T13:51:20.951184+0000","last_clean":"2024-09-13T13:51:20.951184+0000","last_became_active":"2024-09-13T13:51:02.819431+0000","last_became_peered":"2024-09-13T13:51:02.819431+0000","last_unstale":"2024-09-13T13:51:20.951184+0000","last_undegraded":"2024-09-13T13:51:20.951184+0000","last_fullsized":"2024-09-13T13:51:20.951184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:29.904152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039844099999999998,"stat_sum":{"num_bytes":15563913,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":158,"num_read_kb":151087,"num_write":71,"num_write_kb":20945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27682344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":194,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T13:51:18.779543+0000","last_change":"2024-09-13T13:51:09.226997+0000","last_active":"2024-09-13T13:51:18.779543+0000","last_peered":"2024-09-13T13:51:18.779543+0000","last_clean":"2024-09-13T13:51:18.779543+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:51:18.779543+0000","last_undegraded":"2024-09-13T13:51:18.779543+0000","last_fullsized":"2024-09-13T13:51:18.779543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:51:09.226898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.226898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:21.066559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"101'123","reported_seq":289,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-13T13:51:19.885676+0000","last_change":"2024-09-13T13:50:05.074631+0000","last_active":"2024-09-13T13:51:19.885676+0000","last_peered":"2024-09-13T13:51:19.885676+0000","last_clean":"2024-09-13T13:51:19.885676+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:51:19.885676+0000","last_undegraded":"2024-09-13T13:51:19.885676+0000","last_fullsized":"2024-09-13T13:51:19.885676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18478314,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":112,"num_read_kb":5780,"num_write":138,"num_write_kb":44333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"99'172","reported_seq":364,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900224+0000","last_change":"2024-09-13T13:51:16.883549+0000","last_active":"2024-09-13T13:51:20.900224+0000","last_peered":"2024-09-13T13:51:20.900224+0000","last_clean":"2024-09-13T13:51:20.900224+0000","last_became_active":"2024-09-13T13:51:06.832059+0000","last_became_peered":"2024-09-13T13:51:06.832059+0000","last_unstale":"2024-09-13T13:51:20.900224+0000","last_undegraded":"2024-09-13T13:51:20.900224+0000","last_fullsized":"2024-09-13T13:51:20.900224+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T13:51:10.824926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:10.824926+0000","objects_scrubbed":10,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.866369+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0.00048635900000000001,"stat_sum":{"num_bytes":17952805,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":228,"num_read_kb":183440,"num_write":95,"num_write_kb":26693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":33977752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":171,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.875816+0000","last_change":"2024-09-13T13:51:13.190624+0000","last_active":"2024-09-13T13:51:17.875816+0000","last_peered":"2024-09-13T13:51:17.875816+0000","last_clean":"2024-09-13T13:51:17.875816+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:51:17.875816+0000","last_undegraded":"2024-09-13T13:51:17.875816+0000","last_fullsized":"2024-09-13T13:51:17.875816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:13.190561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.190561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.641712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"63'23","reported_seq":336,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.896239+0000","last_change":"2024-09-13T13:51:16.877550+0000","last_active":"2024-09-13T13:51:20.896239+0000","last_peered":"2024-09-13T13:51:20.896239+0000","last_clean":"2024-09-13T13:51:20.896239+0000","last_became_active":"2024-09-13T13:50:06.075820+0000","last_became_peered":"2024-09-13T13:50:06.075820+0000","last_unstale":"2024-09-13T13:51:20.896239+0000","last_undegraded":"2024-09-13T13:51:20.896239+0000","last_fullsized":"2024-09-13T13:51:20.896239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'23","last_scrub_stamp":"2024-09-13T13:51:14.735588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.735588+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:34.170512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000294245,"stat_sum":{"num_bytes":3445402,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":61880,"num_write":15,"num_write_kb":4441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.276617+0000","last_change":"2024-09-13T13:51:01.508987+0000","last_active":"2024-09-13T13:51:17.276617+0000","last_peered":"2024-09-13T13:51:17.276617+0000","last_clean":"2024-09-13T13:51:17.276617+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:51:17.276617+0000","last_undegraded":"2024-09-13T13:51:17.276617+0000","last_fullsized":"2024-09-13T13:51:17.276617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:01.508923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:01.508923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:10.969122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"100'59","reported_seq":368,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902845+0000","last_change":"2024-09-13T13:51:16.877935+0000","last_active":"2024-09-13T13:51:20.902845+0000","last_peered":"2024-09-13T13:51:20.902845+0000","last_clean":"2024-09-13T13:51:20.902845+0000","last_became_active":"2024-09-13T13:50:06.073006+0000","last_became_peered":"2024-09-13T13:50:06.073006+0000","last_unstale":"2024-09-13T13:51:20.902845+0000","last_undegraded":"2024-09-13T13:51:20.902845+0000","last_fullsized":"2024-09-13T13:51:20.902845+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:33.250439+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0028241609999999999,"stat_sum":{"num_bytes":4720896,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":66527,"num_write":30,"num_write_kb":6524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":174,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T13:51:18.779564+0000","last_change":"2024-09-13T13:50:57.984106+0000","last_active":"2024-09-13T13:51:18.779564+0000","last_peered":"2024-09-13T13:51:18.779564+0000","last_clean":"2024-09-13T13:51:18.779564+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:51:18.779564+0000","last_undegraded":"2024-09-13T13:51:18.779564+0000","last_fullsized":"2024-09-13T13:51:18.779564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:57.983997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:50:57.983997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:05.913635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"102'228","reported_seq":732,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.896224+0000","last_change":"2024-09-13T13:51:16.877500+0000","last_active":"2024-09-13T13:51:20.896224+0000","last_peered":"2024-09-13T13:51:20.896224+0000","last_clean":"2024-09-13T13:51:20.896224+0000","last_became_active":"2024-09-13T13:50:54.737668+0000","last_became_peered":"2024-09-13T13:50:54.737668+0000","last_unstale":"2024-09-13T13:51:20.896224+0000","last_undegraded":"2024-09-13T13:51:20.896224+0000","last_fullsized":"2024-09-13T13:51:20.896224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'211","last_scrub_stamp":"2024-09-13T13:51:13.786427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.786427+0000","objects_scrubbed":13,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:25.341250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00030891399999999998,"stat_sum":{"num_bytes":30673693,"num_objects":13,"num_object_clones":7,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":417,"num_read_kb":341987,"num_write":116,"num_write_kb":32597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":29261751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131272+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:51:17.131272+0000","last_peered":"2024-09-13T13:51:17.131272+0000","last_clean":"2024-09-13T13:51:17.131272+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:51:17.131272+0000","last_undegraded":"2024-09-13T13:51:17.131272+0000","last_fullsized":"2024-09-13T13:51:17.131272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:21.129821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":243114044,"num_objects":121,"num_object_clones":71,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":6,"num_read":2830,"num_read_kb":2315921,"num_write":1126,"num_write_kb":307593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":200987241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355418112,"data_stored":465389942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2033,"ondisk_log_size":2033,"up":32,"acting":32,"num_store_stats":15},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81791778,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":416,"num_read_kb":21059,"num_write":542,"num_write_kb":196503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":91329756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183119872,"data_stored":182798462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":498,"ondisk_log_size":498,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":5720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":5857280,"data_stored":6383045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247829,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72272,"kb_used_data":41088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299568,"statfs":{"total":96636764160,"available":96562757632,"internally_reserved":0,"allocated":42074112,"data_stored":48245654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37500,"kb_used_data":10476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334340,"statfs":{"total":96636764160,"available":96598364160,"internally_reserved":0,"allocated":10727424,"data_stored":12012107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97640,"kb_used_data":66456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94274200,"statfs":{"total":96636764160,"available":96536780800,"internally_reserved":0,"allocated":68050944,"data_stored":84403607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74732,"kb_used_data":47708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94297108,"statfs":{"total":96636764160,"available":96560238592,"internally_reserved":0,"allocated":48852992,"data_stored":50069453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41156,"kb_used_data":14132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":14471168,"data_stored":20012141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112504,"kb_used_data":81320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94259336,"statfs":{"total":96636764160,"available":96521560064,"internally_reserved":0,"allocated":83271680,"data_stored":105773715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":667495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61464,"kb_used_data":34440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94310376,"statfs":{"total":96636764160,"available":96573825024,"internally_reserved":0,"allocated":35266560,"data_stored":40802876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41676,"kb_used_data":14652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330164,"statfs":{"total":96636764160,"available":96594087936,"internally_reserved":0,"allocated":15003648,"data_stored":19948764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313238,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67800,"kb_used_data":36616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304040,"statfs":{"total":96636764160,"available":96567336960,"internally_reserved":0,"allocated":37494784,"data_stored":40050052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":925696,"data_stored":635606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38968,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332872,"statfs":{"total":96636764160,"available":96596860928,"internally_reserved":0,"allocated":12230656,"data_stored":17798522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69548,"kb_used_data":38364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302292,"statfs":{"total":96636764160,"available":96565547008,"internally_reserved":0,"allocated":39284736,"data_stored":46029360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50424,"kb_used_data":23400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94321416,"statfs":{"total":96636764160,"available":96585129984,"internally_reserved":0,"allocated":23961600,"data_stored":24191500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247830,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101460,"kb_used_data":70276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270380,"statfs":{"total":96636764160,"available":96532869120,"internally_reserved":0,"allocated":71962624,"data_stored":83276844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39915520,"data_stored":39837965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23126016,"data_stored":23088939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":18714624,"data_stored":18678541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17924096,"data_stored":17893610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":18478314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24674304,"data_stored":24633643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":20033536,"data_stored":20001280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":18655977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11214848,"data_stored":17149284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68087808,"data_stored":88885017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7933952,"data_stored":9560289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15949824,"data_stored":23739242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15708160,"data_stored":22646240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":85065728,"data_stored":109412764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5275648,"data_stored":5856532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53075968,"data_stored":64739389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15740928,"data_stored":21645888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":9740288,"data_stored":11360071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":15089664,"data_stored":20858003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13365248,"data_stored":18158272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":20996096,"data_stored":27580121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":5142853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:22.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:22.616 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:22.615+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:22.618 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:22.618 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:22.618 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:22.618 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:22.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:22.717 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:22.715+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:22.731 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:22.731 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:22.731 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:22.731 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:22.731 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:22.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:22.818 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:22.817+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 92.655444302999996, 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:51:22.885 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:51:22.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.150706120999999, 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:51:22.892 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:51:22.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.898 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.150656808999997, 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.899 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:51:22.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:51:22.904 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.150641105000005, 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.905 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:51:22.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.150613429000003, 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:51:22.911 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:51:22.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:51:22.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.115395831000001, 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.918 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:51:22.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:51:22.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:51:22.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:22.924 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.108326848999994, 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:51:22.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.073788386999993, 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:51:22.931 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:22.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:51:22.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:51:22.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.937 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.066158544999993, 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.938 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:51:22.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:51:22.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.062782173000002, 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.944 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:51:22.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:51:22.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.959922892999998, 2024-09-13T13:51:22.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:22.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:51:22.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.956 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.748899352999999, 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.957 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.737384070000004, 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.636075601999998, 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:51:22.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:51:22.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:22.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:51:22.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.636011254000003, 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:22.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:51:22.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:22.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.977 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 42.635983191999998, 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:51:22.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:51:22.983 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.634301301999997, 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:22.984 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:51:22.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:22.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:22.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1811 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.627248281999996, 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030548815999999999, 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:22.992 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1811 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:22.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255828+0000", 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:22.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255835+0000", 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2119999999999997e-06 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256027+0000", 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018704600000000001 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256075+0000", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260766+0000", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046910119999999996 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260793+0000", 2024-09-13T13:51:22.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6594999999999999e-05 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280369+0000", 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019576296 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280418+0000", 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9221999999999997e-05 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:22.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286090+0000", 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005671873 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286195+0000", 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010464299999999999 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286374+0000", 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179118 2024-09-13T13:51:22.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.194773093999999, 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:51:23.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:51:23.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:23.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.066137339000001, 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:23.005 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:23.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:51:23.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:23.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:51:23.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:23.011 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:23.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:23.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:23.011 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:23.011 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:23.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:51:23.013 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:23.011+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:23.014 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:51:23.015 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:51:23.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:23.018 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T13:51:22.379093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":325496209,"num_objects":203,"num_object_clones":71,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":6,"num_read":3332,"num_read_kb":2337053,"num_write":1815,"num_write_kb":506770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":292907403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2646,"ondisk_log_size":2646,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":955816,"kb_used_data":498472,"kb_used_omap":24,"kb_used_meta":457063,"kb_avail":1508993624,"statfs":{"total":1546188226560,"available":1545209470976,"internally_reserved":0,"allocated":510435328,"data_stored":600300741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25519,"internal_metadata":468032593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13152948,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":-1,"num_read":98,"num_read_kb":78403,"num_write":44,"num_write_kb":10881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000859"},"pg_stats":[{"pgid":"4.f","version":"104'170","reported_seq":524,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.972351+0000","last_change":"2024-09-13T13:51:17.277078+0000","last_active":"2024-09-13T13:51:20.972351+0000","last_peered":"2024-09-13T13:51:20.972351+0000","last_clean":"2024-09-13T13:51:20.972351+0000","last_became_active":"2024-09-13T13:50:06.073131+0000","last_became_peered":"2024-09-13T13:50:06.073131+0000","last_unstale":"2024-09-13T13:51:20.972351+0000","last_undegraded":"2024-09-13T13:51:20.972351+0000","last_fullsized":"2024-09-13T13:51:20.972351+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000349367,"stat_sum":{"num_bytes":12634648,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":240,"num_read_kb":186240,"num_write":86,"num_write_kb":21450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.e","version":"100'196","reported_seq":584,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.874564+0000","last_change":"2024-09-13T13:51:16.874510+0000","last_active":"2024-09-13T13:51:17.874564+0000","last_peered":"2024-09-13T13:51:17.874564+0000","last_clean":"2024-09-13T13:51:17.874564+0000","last_became_active":"2024-09-13T13:50:06.072867+0000","last_became_peered":"2024-09-13T13:50:06.072867+0000","last_unstale":"2024-09-13T13:51:17.874564+0000","last_undegraded":"2024-09-13T13:51:17.874564+0000","last_fullsized":"2024-09-13T13:51:17.874564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'178","last_scrub_stamp":"2024-09-13T13:51:13.109829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.109829+0000","objects_scrubbed":9,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:18.981431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0017295500000000001,"stat_sum":{"num_bytes":19057900,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":336,"num_read_kb":251766,"num_write":104,"num_write_kb":27450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.d","version":"102'38","reported_seq":341,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900989+0000","last_change":"2024-09-13T13:51:16.876002+0000","last_active":"2024-09-13T13:51:20.900989+0000","last_peered":"2024-09-13T13:51:20.900989+0000","last_clean":"2024-09-13T13:51:20.900989+0000","last_became_active":"2024-09-13T13:50:06.075524+0000","last_became_peered":"2024-09-13T13:50:06.075524+0000","last_unstale":"2024-09-13T13:51:20.900989+0000","last_undegraded":"2024-09-13T13:51:20.900989+0000","last_fullsized":"2024-09-13T13:51:20.900989+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034429399999999998,"stat_sum":{"num_bytes":5276353,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":43420,"num_write":21,"num_write_kb":5134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.c","version":"104'174","reported_seq":588,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.926519+0000","last_change":"2024-09-13T13:51:16.879999+0000","last_active":"2024-09-13T13:51:20.926519+0000","last_peered":"2024-09-13T13:51:20.926519+0000","last_clean":"2024-09-13T13:51:20.926519+0000","last_became_active":"2024-09-13T13:50:28.729220+0000","last_became_peered":"2024-09-13T13:50:28.729220+0000","last_unstale":"2024-09-13T13:51:20.926519+0000","last_undegraded":"2024-09-13T13:51:20.926519+0000","last_fullsized":"2024-09-13T13:51:20.926519+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'162","last_scrub_stamp":"2024-09-13T13:51:12.406192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:12.406192+0000","objects_scrubbed":8,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:23.557831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.004311283,"stat_sum":{"num_bytes":19657354,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":255,"num_read_kb":231014,"num_write":97,"num_write_kb":24394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11131276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.b","version":"99'199","reported_seq":516,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876472+0000","last_change":"2024-09-13T13:51:16.875325+0000","last_active":"2024-09-13T13:51:17.876472+0000","last_peered":"2024-09-13T13:51:17.876472+0000","last_clean":"2024-09-13T13:51:17.876472+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:51:17.876472+0000","last_undegraded":"2024-09-13T13:51:17.876472+0000","last_fullsized":"2024-09-13T13:51:17.876472+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:24.347511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031213399999999999,"stat_sum":{"num_bytes":30278287,"num_objects":15,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":163,"num_read_kb":104738,"num_write":122,"num_write_kb":32130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.a","version":"99'137","reported_seq":573,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131764+0000","last_change":"2024-09-13T13:51:17.131913+0000","last_active":"2024-09-13T13:51:17.131764+0000","last_peered":"2024-09-13T13:51:17.131764+0000","last_clean":"2024-09-13T13:51:17.131764+0000","last_became_active":"2024-09-13T13:51:02.821659+0000","last_became_peered":"2024-09-13T13:51:02.821659+0000","last_unstale":"2024-09-13T13:51:17.131764+0000","last_undegraded":"2024-09-13T13:51:17.131764+0000","last_fullsized":"2024-09-13T13:51:17.131764+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414707,"stat_sum":{"num_bytes":24443983,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":260,"num_read_kb":253969,"num_write":75,"num_write_kb":23136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":34232380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.9","version":"95'21","reported_seq":296,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.431579+0000","last_change":"2024-09-13T13:51:17.159367+0000","last_active":"2024-09-13T13:51:17.431579+0000","last_peered":"2024-09-13T13:51:17.431579+0000","last_clean":"2024-09-13T13:51:17.431579+0000","last_became_active":"2024-09-13T13:50:12.197810+0000","last_became_peered":"2024-09-13T13:50:12.197810+0000","last_unstale":"2024-09-13T13:51:17.431579+0000","last_undegraded":"2024-09-13T13:51:17.431579+0000","last_fullsized":"2024-09-13T13:51:17.431579+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032133200000000002,"stat_sum":{"num_bytes":2571471,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":40643,"num_write":14,"num_write_kb":3605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5],"acting":[15,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.8","version":"101'108","reported_seq":419,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902697+0000","last_change":"2024-09-13T13:51:17.277230+0000","last_active":"2024-09-13T13:51:20.902697+0000","last_peered":"2024-09-13T13:51:20.902697+0000","last_clean":"2024-09-13T13:51:20.902697+0000","last_became_active":"2024-09-13T13:50:06.075916+0000","last_became_peered":"2024-09-13T13:50:06.075916+0000","last_unstale":"2024-09-13T13:51:20.902697+0000","last_undegraded":"2024-09-13T13:51:20.902697+0000","last_fullsized":"2024-09-13T13:51:20.902697+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:23.021997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043106399999999999,"stat_sum":{"num_bytes":11823525,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":107,"num_read_kb":82752,"num_write":55,"num_write_kb":14502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"4.5","version":"101'175","reported_seq":427,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.901166+0000","last_change":"2024-09-13T13:51:16.883364+0000","last_active":"2024-09-13T13:51:20.901166+0000","last_peered":"2024-09-13T13:51:20.901166+0000","last_clean":"2024-09-13T13:51:20.901166+0000","last_became_active":"2024-09-13T13:51:05.828626+0000","last_became_peered":"2024-09-13T13:51:05.828626+0000","last_unstale":"2024-09-13T13:51:20.901166+0000","last_undegraded":"2024-09-13T13:51:20.901166+0000","last_fullsized":"2024-09-13T13:51:20.901166+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'163","last_scrub_stamp":"2024-09-13T13:51:14.489895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.489895+0000","objects_scrubbed":12,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:28.143616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00034909699999999998,"stat_sum":{"num_bytes":21062400,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":171,"num_read_kb":129526,"num_write":94,"num_write_kb":28485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":40416780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":271,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900090+0000","last_change":"2024-09-13T13:51:09.467323+0000","last_active":"2024-09-13T13:51:20.900090+0000","last_peered":"2024-09-13T13:51:20.900090+0000","last_clean":"2024-09-13T13:51:20.900090+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:51:20.900090+0000","last_undegraded":"2024-09-13T13:51:20.900090+0000","last_fullsized":"2024-09-13T13:51:20.900090+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:51:09.467249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.467249+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:36.764015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":171,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876473+0000","last_change":"2024-09-13T13:51:13.789246+0000","last_active":"2024-09-13T13:51:17.876473+0000","last_peered":"2024-09-13T13:51:17.876473+0000","last_clean":"2024-09-13T13:51:17.876473+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:51:17.876473+0000","last_undegraded":"2024-09-13T13:51:17.876473+0000","last_fullsized":"2024-09-13T13:51:17.876473+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:51:13.789157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.789157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:16.372966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"95'123","reported_seq":410,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876473+0000","last_change":"2024-09-13T13:50:51.191971+0000","last_active":"2024-09-13T13:51:17.876473+0000","last_peered":"2024-09-13T13:51:17.876473+0000","last_clean":"2024-09-13T13:51:17.876473+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:51:17.876473+0000","last_undegraded":"2024-09-13T13:51:17.876473+0000","last_fullsized":"2024-09-13T13:51:17.876473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:19.528621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20001280,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":100,"num_read_kb":4842,"num_write":132,"num_write_kb":52676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"100'96","reported_seq":469,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.664703+0000","last_change":"2024-09-13T13:51:17.277204+0000","last_active":"2024-09-13T13:51:17.664703+0000","last_peered":"2024-09-13T13:51:17.664703+0000","last_clean":"2024-09-13T13:51:17.664703+0000","last_became_active":"2024-09-13T13:51:00.088253+0000","last_became_peered":"2024-09-13T13:51:00.088253+0000","last_unstale":"2024-09-13T13:51:17.664703+0000","last_undegraded":"2024-09-13T13:51:17.664703+0000","last_fullsized":"2024-09-13T13:51:17.664703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:21.234859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043333299999999999,"stat_sum":{"num_bytes":8233881,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":146,"num_read_kb":129809,"num_write":55,"num_write_kb":15323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14889074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131190+0000","last_change":"2024-09-13T13:49:52.242339+0000","last_active":"2024-09-13T13:51:17.131190+0000","last_peered":"2024-09-13T13:51:17.131190+0000","last_clean":"2024-09-13T13:51:17.131190+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:51:17.131190+0000","last_undegraded":"2024-09-13T13:51:17.131190+0000","last_fullsized":"2024-09-13T13:51:17.131190+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"104'132","reported_seq":290,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.967858+0000","last_change":"2024-09-13T13:50:05.123338+0000","last_active":"2024-09-13T13:51:20.967858+0000","last_peered":"2024-09-13T13:51:20.967858+0000","last_clean":"2024-09-13T13:51:20.967858+0000","last_became_active":"2024-09-13T13:50:05.123078+0000","last_became_peered":"2024-09-13T13:50:05.123078+0000","last_unstale":"2024-09-13T13:51:20.967858+0000","last_undegraded":"2024-09-13T13:51:20.967858+0000","last_fullsized":"2024-09-13T13:51:20.967858+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:27.240224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633643,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":108,"num_read_kb":8112,"num_write":145,"num_write_kb":53772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"99'122","reported_seq":461,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.875824+0000","last_change":"2024-09-13T13:51:17.274010+0000","last_active":"2024-09-13T13:51:17.875824+0000","last_peered":"2024-09-13T13:51:17.875824+0000","last_clean":"2024-09-13T13:51:17.875824+0000","last_became_active":"2024-09-13T13:50:29.051980+0000","last_became_peered":"2024-09-13T13:50:29.051980+0000","last_unstale":"2024-09-13T13:51:17.875824+0000","last_undegraded":"2024-09-13T13:51:17.875824+0000","last_fullsized":"2024-09-13T13:51:17.875824+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:23.929566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048694199999999997,"stat_sum":{"num_bytes":15717533,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":106,"num_read_kb":57123,"num_write":76,"num_write_kb":20784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9395884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"1"},{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":170,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.876472+0000","last_change":"2024-09-13T13:51:11.801324+0000","last_active":"2024-09-13T13:51:17.876472+0000","last_peered":"2024-09-13T13:51:17.876472+0000","last_clean":"2024-09-13T13:51:17.876472+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:51:17.876472+0000","last_undegraded":"2024-09-13T13:51:17.876472+0000","last_fullsized":"2024-09-13T13:51:17.876472+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:11.801231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:11.801231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.494865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"98'120","reported_seq":164,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902480+0000","last_change":"2024-09-13T13:50:50.914930+0000","last_active":"2024-09-13T13:51:20.902480+0000","last_peered":"2024-09-13T13:51:20.902480+0000","last_clean":"2024-09-13T13:51:20.902480+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:51:20.902480+0000","last_undegraded":"2024-09-13T13:51:20.902480+0000","last_fullsized":"2024-09-13T13:51:20.902480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18678541,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":96,"num_read_kb":2325,"num_write":127,"num_write_kb":45722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"104'115","reported_seq":130,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.951184+0000","last_change":"2024-09-13T13:51:16.883427+0000","last_active":"2024-09-13T13:51:20.951184+0000","last_peered":"2024-09-13T13:51:20.951184+0000","last_clean":"2024-09-13T13:51:20.951184+0000","last_became_active":"2024-09-13T13:51:02.819431+0000","last_became_peered":"2024-09-13T13:51:02.819431+0000","last_unstale":"2024-09-13T13:51:20.951184+0000","last_undegraded":"2024-09-13T13:51:20.951184+0000","last_fullsized":"2024-09-13T13:51:20.951184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:29.904152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039844099999999998,"stat_sum":{"num_bytes":15563913,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":158,"num_read_kb":151087,"num_write":71,"num_write_kb":20945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27682344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":194,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T13:51:18.779543+0000","last_change":"2024-09-13T13:51:09.226997+0000","last_active":"2024-09-13T13:51:18.779543+0000","last_peered":"2024-09-13T13:51:18.779543+0000","last_clean":"2024-09-13T13:51:18.779543+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:51:18.779543+0000","last_undegraded":"2024-09-13T13:51:18.779543+0000","last_fullsized":"2024-09-13T13:51:18.779543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:51:09.226898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.226898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:21.066559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"101'123","reported_seq":289,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-13T13:51:19.885676+0000","last_change":"2024-09-13T13:50:05.074631+0000","last_active":"2024-09-13T13:51:19.885676+0000","last_peered":"2024-09-13T13:51:19.885676+0000","last_clean":"2024-09-13T13:51:19.885676+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:51:19.885676+0000","last_undegraded":"2024-09-13T13:51:19.885676+0000","last_fullsized":"2024-09-13T13:51:19.885676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18478314,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":112,"num_read_kb":5780,"num_write":138,"num_write_kb":44333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"99'172","reported_seq":364,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.900224+0000","last_change":"2024-09-13T13:51:16.883549+0000","last_active":"2024-09-13T13:51:20.900224+0000","last_peered":"2024-09-13T13:51:20.900224+0000","last_clean":"2024-09-13T13:51:20.900224+0000","last_became_active":"2024-09-13T13:51:06.832059+0000","last_became_peered":"2024-09-13T13:51:06.832059+0000","last_unstale":"2024-09-13T13:51:20.900224+0000","last_undegraded":"2024-09-13T13:51:20.900224+0000","last_fullsized":"2024-09-13T13:51:20.900224+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T13:51:10.824926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:10.824926+0000","objects_scrubbed":10,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.866369+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0.00048635900000000001,"stat_sum":{"num_bytes":17952805,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":228,"num_read_kb":183440,"num_write":95,"num_write_kb":26693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":33977752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":171,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.875816+0000","last_change":"2024-09-13T13:51:13.190624+0000","last_active":"2024-09-13T13:51:17.875816+0000","last_peered":"2024-09-13T13:51:17.875816+0000","last_clean":"2024-09-13T13:51:17.875816+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:51:17.875816+0000","last_undegraded":"2024-09-13T13:51:17.875816+0000","last_fullsized":"2024-09-13T13:51:17.875816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:13.190561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.190561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.641712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"63'23","reported_seq":336,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.896239+0000","last_change":"2024-09-13T13:51:16.877550+0000","last_active":"2024-09-13T13:51:20.896239+0000","last_peered":"2024-09-13T13:51:20.896239+0000","last_clean":"2024-09-13T13:51:20.896239+0000","last_became_active":"2024-09-13T13:50:06.075820+0000","last_became_peered":"2024-09-13T13:50:06.075820+0000","last_unstale":"2024-09-13T13:51:20.896239+0000","last_undegraded":"2024-09-13T13:51:20.896239+0000","last_fullsized":"2024-09-13T13:51:20.896239+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'23","last_scrub_stamp":"2024-09-13T13:51:14.735588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.735588+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:34.170512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000294245,"stat_sum":{"num_bytes":3445402,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":61880,"num_write":15,"num_write_kb":4441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.276617+0000","last_change":"2024-09-13T13:51:01.508987+0000","last_active":"2024-09-13T13:51:17.276617+0000","last_peered":"2024-09-13T13:51:17.276617+0000","last_clean":"2024-09-13T13:51:17.276617+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:51:17.276617+0000","last_undegraded":"2024-09-13T13:51:17.276617+0000","last_fullsized":"2024-09-13T13:51:17.276617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:01.508923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:01.508923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:10.969122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"100'59","reported_seq":368,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.902845+0000","last_change":"2024-09-13T13:51:16.877935+0000","last_active":"2024-09-13T13:51:20.902845+0000","last_peered":"2024-09-13T13:51:20.902845+0000","last_clean":"2024-09-13T13:51:20.902845+0000","last_became_active":"2024-09-13T13:50:06.073006+0000","last_became_peered":"2024-09-13T13:50:06.073006+0000","last_unstale":"2024-09-13T13:51:20.902845+0000","last_undegraded":"2024-09-13T13:51:20.902845+0000","last_fullsized":"2024-09-13T13:51:20.902845+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:50:05.059988+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:33.250439+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0028241609999999999,"stat_sum":{"num_bytes":4720896,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":66527,"num_write":30,"num_write_kb":6524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":174,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-13T13:51:18.779564+0000","last_change":"2024-09-13T13:50:57.984106+0000","last_active":"2024-09-13T13:51:18.779564+0000","last_peered":"2024-09-13T13:51:18.779564+0000","last_clean":"2024-09-13T13:51:18.779564+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:51:18.779564+0000","last_undegraded":"2024-09-13T13:51:18.779564+0000","last_fullsized":"2024-09-13T13:51:18.779564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:57.983997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:50:57.983997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:05.913635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"102'228","reported_seq":732,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-13T13:51:20.896224+0000","last_change":"2024-09-13T13:51:16.877500+0000","last_active":"2024-09-13T13:51:20.896224+0000","last_peered":"2024-09-13T13:51:20.896224+0000","last_clean":"2024-09-13T13:51:20.896224+0000","last_became_active":"2024-09-13T13:50:54.737668+0000","last_became_peered":"2024-09-13T13:50:54.737668+0000","last_unstale":"2024-09-13T13:51:20.896224+0000","last_undegraded":"2024-09-13T13:51:20.896224+0000","last_fullsized":"2024-09-13T13:51:20.896224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'211","last_scrub_stamp":"2024-09-13T13:51:13.786427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.786427+0000","objects_scrubbed":13,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:25.341250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00030891399999999998,"stat_sum":{"num_bytes":30673693,"num_objects":13,"num_object_clones":7,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":417,"num_read_kb":341987,"num_write":116,"num_write_kb":32597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":29261751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"2"},{"start":"1d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-13T13:51:17.131272+0000","last_change":"2024-09-13T13:49:52.242335+0000","last_active":"2024-09-13T13:51:17.131272+0000","last_peered":"2024-09-13T13:51:17.131272+0000","last_clean":"2024-09-13T13:51:17.131272+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:51:17.131272+0000","last_undegraded":"2024-09-13T13:51:17.131272+0000","last_fullsized":"2024-09-13T13:51:17.131272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:49:49.210716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:51:21.129821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":243114044,"num_objects":121,"num_object_clones":71,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":6,"num_read":2830,"num_read_kb":2315921,"num_write":1126,"num_write_kb":307593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":200987241,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355418112,"data_stored":465389942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2033,"ondisk_log_size":2033,"up":32,"acting":32,"num_store_stats":15},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81791778,"num_objects":78,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":416,"num_read_kb":21059,"num_write":542,"num_write_kb":196503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":91329756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183119872,"data_stored":182798462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":498,"ondisk_log_size":498,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":5720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":5857280,"data_stored":6383045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247829,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72272,"kb_used_data":41088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299568,"statfs":{"total":96636764160,"available":96562757632,"internally_reserved":0,"allocated":42074112,"data_stored":48245654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37500,"kb_used_data":10476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334340,"statfs":{"total":96636764160,"available":96598364160,"internally_reserved":0,"allocated":10727424,"data_stored":12012107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97640,"kb_used_data":66456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94274200,"statfs":{"total":96636764160,"available":96536780800,"internally_reserved":0,"allocated":68050944,"data_stored":84403607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74732,"kb_used_data":47708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94297108,"statfs":{"total":96636764160,"available":96560238592,"internally_reserved":0,"allocated":48852992,"data_stored":50069453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41156,"kb_used_data":14132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":14471168,"data_stored":20012141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,6,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112504,"kb_used_data":81320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94259336,"statfs":{"total":96636764160,"available":96521560064,"internally_reserved":0,"allocated":83271680,"data_stored":105773715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":667495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61464,"kb_used_data":34440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94310376,"statfs":{"total":96636764160,"available":96573825024,"internally_reserved":0,"allocated":35266560,"data_stored":40802876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41676,"kb_used_data":14652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330164,"statfs":{"total":96636764160,"available":96594087936,"internally_reserved":0,"allocated":15003648,"data_stored":19948764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313238,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67800,"kb_used_data":36616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304040,"statfs":{"total":96636764160,"available":96567336960,"internally_reserved":0,"allocated":37494784,"data_stored":40050052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":925696,"data_stored":635606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38968,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332872,"statfs":{"total":96636764160,"available":96596860928,"internally_reserved":0,"allocated":12230656,"data_stored":17798522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69548,"kb_used_data":38364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302292,"statfs":{"total":96636764160,"available":96565547008,"internally_reserved":0,"allocated":39284736,"data_stored":46029360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50424,"kb_used_data":23400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94321416,"statfs":{"total":96636764160,"available":96585129984,"internally_reserved":0,"allocated":23961600,"data_stored":24191500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247830,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101460,"kb_used_data":70276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270380,"statfs":{"total":96636764160,"available":96532869120,"internally_reserved":0,"allocated":71962624,"data_stored":83276844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39915520,"data_stored":39837965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":23126016,"data_stored":23088939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":18714624,"data_stored":18678541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17924096,"data_stored":17893610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":18478314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24674304,"data_stored":24633643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":20033536,"data_stored":20001280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":18655977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11214848,"data_stored":17149284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68087808,"data_stored":88885017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7933952,"data_stored":9560289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15949824,"data_stored":23739242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15708160,"data_stored":22646240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":85065728,"data_stored":109412764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5275648,"data_stored":5856532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":53075968,"data_stored":64739389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":15740928,"data_stored":21645888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":9740288,"data_stored":11360071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":15089664,"data_stored":20858003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13365248,"data_stored":18158272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":20996096,"data_stored":27580121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":5142853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:23.020 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:51:23.020 INFO:tasks.rados.rados.0.smithi159.stdout:1438: done (13 left) 2024-09-13T13:51:23.020 INFO:tasks.rados.rados.0.smithi159.stdout:1439: read oid 1 snap -1 2024-09-13T13:51:23.020 INFO:tasks.rados.rados.0.smithi159.stdout:1439: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1440: rollback oid 20 current snap is 30 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 20 to 16 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1428: got (22) Invalid argument 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1429: finishing rollback tid 1 to smithi15938319-30 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 179 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1425: finishing write tid 1 to smithi15938319-19 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1428: done (14 left) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1429: done (13 left) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1430: done (12 left) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1432: done (11 left) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1441: read oid 29 snap -1 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1441: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:23.021 INFO:tasks.rados.rados.0.smithi159.stdout:1442: set_chunk oid 45 target oid 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1442: set_chunk oid 45 offset: 1245696 length: 1389568 target oid 2abb7c9824490d58538bd19b531c0d3becffba183c014811388cd80bdd8c2ca5 offset: 0 snap -1 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1443: read oid 26 snap -1 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1443: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1444: read oid 6 snap -1 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1444: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1445: delete oid 37 current snap is 30 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1431: got (0) Success 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 199 (ObjNum 232 snap 26 seq_num 232) dirty exists 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1425: finishing write tid 2 to smithi15938319-19 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1425: finishing write tid 3 to smithi15938319-19 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1425: finishing write tid 4 to smithi15938319-19 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1425: oid 19 updating version 0 to 129 2024-09-13T13:51:23.022 INFO:tasks.rados.rados.0.smithi159.stdout:1425: oid 19 updating version 129 to 130 2024-09-13T13:51:23.023 INFO:tasks.rados.rados.0.smithi159.stdout:1425: oid 19 version 130 is already newer than 128 2024-09-13T13:51:23.023 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 130 (ObjNum 260 snap 30 seq_num 260) dirty exists 2024-09-13T13:51:23.023 INFO:tasks.rados.rados.0.smithi159.stdout:1425: left oid 19 (ObjNum 260 snap 30 seq_num 260) 2024-09-13T13:51:23.023 INFO:tasks.rados.rados.0.smithi159.stdout:1427: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:23.023 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 861728 2024-09-13T13:51:23.069 INFO:tasks.rados.rados.0.smithi159.stdout:1426: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:23.069 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:23.114 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:23.113+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:23.126 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.127 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:23.127 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:23.127 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.135 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:23.215 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:23.214+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:23.239 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.239 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:23.239 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:23.239 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:23.239 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.247 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:23.257 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3089907 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1433: finishing set_chunk to oid 48 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 185 (ObjNum 219 snap 24 seq_num 219) dirty exists 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1434: finishing write tid 1 to smithi15938319-8 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1434: finishing write tid 2 to smithi15938319-8 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1425: done (15 left) 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1426: done (14 left) 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1427: done (13 left) 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1431: done (12 left) 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1433: done (11 left) 2024-09-13T13:51:23.272 INFO:tasks.rados.rados.0.smithi159.stdout:1445: done (10 left) 2024-09-13T13:51:23.273 INFO:tasks.rados.rados.0.smithi159.stdout:1446: snap_remove snap 17 2024-09-13T13:51:23.316 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:23.314+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:23.362 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.362 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:23.362 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.628127904999999, 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.363 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.364 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.365 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:51:23.366 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.367 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.368 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.369 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.628050504000001, 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.370 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:51:23.371 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:51:23.372 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.373 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.374 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.375 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.627897966000006, 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.376 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:23.377 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.378 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:51:23.379 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.380 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.381 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.627791927000004, 2024-09-13T13:51:23.382 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:23.383 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:23.384 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:51:23.385 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:51:23.386 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.387 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:51:23.388 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:23.389 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.619452848999998, 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.390 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.391 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.392 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:51:23.393 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.394 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.592086139000003, 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.395 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.396 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.397 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:51:23.398 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.399 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:51:23.400 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.401 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.590319541, 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.402 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.585440199000004, 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.408 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.409 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.410 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:219 3.2 e41/33 3:650cb820:::smithi15938319-20 low_tier:head v 41'6, mlcod=41'3)", 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.578349215000003, 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02650433, 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781965+0000", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.452e-06 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789891+0000", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079261269999999998 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789958+0000", 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6587999999999993e-05 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789965+0000", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790010+0000", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4592000000000003e-05 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790030+0000", 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0630000000000001e-05 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808419+0000", 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018388315999999998 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808468+0000", 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9168000000000001e-05 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.574928474000004, 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:51:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:227 3.2 e41/33 3:626302dd:::smithi15938319-28 low_tier:head v 41'7, mlcod=41'3)", 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.570416988000005, 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.024242393000000001, 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789896+0000", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789897+0000", 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8999999999999995e-07 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795632+0000", 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057353760000000004 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795680+0000", 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8420000000000001e-05 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795692+0000", 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1452e-05 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795737+0000", 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5596000000000002e-05 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795751+0000", 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3127e-05 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814104+0000", 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018353103999999999 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814138+0000", 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4527999999999999e-05 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.550671183999995, 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:51:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.547786138999996, 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:51:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.545582113999998, 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.442 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:23.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:51:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:51:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:51:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.53859147, 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:51:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:51:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.433599856000001, 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:51:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.610009994999999, 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:51:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:51:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.609977205, 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:51:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.494217698, 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:51:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.569615009, 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:51:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:51:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.481 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.482 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:23.483 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:23.482+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:23.582 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:23.582+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:23.627 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.627 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:23.627 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:23.627 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.635 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:23.683 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:23.682+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1446: done (10 left) 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1447: snap_create 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1434: finishing write tid 3 to smithi15938319-8 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1434: finishing write tid 4 to smithi15938319-8 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1434: oid 8 updating version 0 to 121 2024-09-13T13:51:23.734 INFO:tasks.rados.rados.0.smithi159.stdout:1434: oid 8 version 121 is already newer than 120 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1434: oid 8 updating version 121 to 122 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 122 (ObjNum 261 snap 30 seq_num 261) dirty exists 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1434: left oid 8 (ObjNum 261 snap 30 seq_num 261) 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1435: finishing set_chunk to oid 16 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 232 (ObjNum 259 snap 30 seq_num 259) dirty exists 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: finishing write tid 1 to smithi15938319-47 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: finishing write tid 2 to smithi15938319-47 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: finishing write tid 3 to smithi15938319-47 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: finishing write tid 4 to smithi15938319-47 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: oid 47 updating version 0 to 144 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: oid 47 version 144 is already newer than 142 2024-09-13T13:51:23.735 INFO:tasks.rados.rados.0.smithi159.stdout:1436: oid 47 version 144 is already newer than 143 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 144 (ObjNum 262 snap 30 seq_num 262) dirty exists 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1436: left oid 47 (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: finishing write tid 1 to smithi15938319-12 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: finishing write tid 2 to smithi15938319-12 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: finishing write tid 3 to smithi15938319-12 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: finishing write tid 4 to smithi15938319-12 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: oid 12 updating version 0 to 176 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: oid 12 updating version 176 to 179 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: oid 12 updating version 179 to 180 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 180 (ObjNum 263 snap 30 seq_num 263) dirty exists 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: left oid 12 (ObjNum 263 snap 30 seq_num 263) 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1440: finishing rollback tid 0 to smithi15938319-20 2024-09-13T13:51:23.736 INFO:tasks.rados.rados.0.smithi159.stdout:1440: finishing rollback tid 1 to smithi15938319-20 2024-09-13T13:51:23.737 INFO:tasks.rados.rados.0.smithi159.stdout:1440: finishing rollback tid 2 to smithi15938319-20 2024-09-13T13:51:23.737 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 104 (ObjNum 131 snap 9 seq_num 131) dirty exists 2024-09-13T13:51:23.737 INFO:tasks.rados.rados.0.smithi159.stdout:1439: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:23.737 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1566647 2024-09-13T13:51:23.740 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.740 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:23.740 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:23.740 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:23.741 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.742 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:23.742+0000 7faeaa210640 -1 osd.8 106 handle_osd_map injecting map crc failure 2024-09-13T13:51:23.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:23.759 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:23.787 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:23.787+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:23.789 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:23.788+0000 7f9b0c553640 -1 osd.12 106 handle_osd_map injecting map crc failure 2024-09-13T13:51:23.821 INFO:tasks.rados.rados.0.smithi159.stdout:1441: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:23.865 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:23.865 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.122264829000002, 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.866 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:51:23.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.116683570000006, 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:51:23.871 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:51:23.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:23.876 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.113706958999998, 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:51:23.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:51:23.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.111732787999998, 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:51:23.881 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:23.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:51:23.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.077973686999997, 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.886 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:51:23.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:51:23.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.073002265, 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.891 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:51:23.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:51:23.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.069100973999994, 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.896 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.900 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.029062905999993, 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:51:23.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.905 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.023887063999993, 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:51:23.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:51:23.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.910 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.947935986000005, 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:51:23.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:51:23.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.943135091000002, 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.917 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:23.922 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.788496253999995, 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:23.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:51:23.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:51:23.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.665615101, 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:51:23.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:51:23.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:51:23.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:23.935 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.608634768000002, 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:51:23.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:51:23.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.603321919000003, 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.942 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:51:23.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.947 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 42.575543412000002, 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:23.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:51:23.951 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 28.066903156999999, 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:23.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:51:23.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 1.952875897, 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.957 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:51:23.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:51:23.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:51:23.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.963 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3176 4.6 4:6ce02a65:::smithi15938319-20:head [rollback 12] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.95531009600000005, 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080956770000000008, 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3176 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908270+0000", 2024-09-13T13:51:23.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4499999999999996e-07 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908274+0000", 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7929999999999999e-06 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908282+0000", 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.276e-06 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908286+0000", 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1930000000000003e-06 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908382+0000", 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5829999999999996e-05 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908389+0000", 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4219999999999997e-06 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912301+0000", 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039123059999999999 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912393+0000", 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1948999999999997e-05 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912525+0000", 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013263500000000001 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914618+0000", 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020925269999999998 2024-09-13T13:51:23.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916287+0000", 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001668803 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916323+0000", 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6715000000000003e-05 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916365+0000", 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1483000000000002e-05 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 0.95527353800000003, 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:23.971 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:23.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:51:23.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:23.979 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:23.979 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:23.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:23.981 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2886923 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1584521 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:1442: finishing set_chunk to oid 45 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 1245696 length: 1389568 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 0 (ObjNum 86 snap 5 seq_num 86) dirty exists 2024-09-13T13:51:23.982 INFO:tasks.rados.rados.0.smithi159.stdout:1443: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:23.982 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:23.979+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:24.003 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:24.041 INFO:tasks.rados.rados.0.smithi159.stdout:1444: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:51:24.041 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:24.082 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:24.081+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:24.090 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:24.107 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:24.107 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:24.107 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:24.107 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:24.114 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:24.116 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:24.137 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:24.158 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:24.159 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:24.179 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:24.180 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:24.183 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:24.182+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:24.200 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:24.200 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:24.222 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:24.243 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:24.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:24.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:24.284 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:24.306 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:24.312 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:24.311+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:24.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:24.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:24.349 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:24.349 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:24.369 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:24.370 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:24.370 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:24.380 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:24.385 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:24.385+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:24.391 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:24.391 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:24.412+0000 7fac0ffff640 0 -- 172.21.15.27:0/4019377118 >> v2:172.21.15.122:6808/1255553857 conn(0x7fac18221ec0 msgr2=0x7fac182242b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:24.486 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:24.485+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:24.523 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:24.532 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:24.586 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:24.586+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:24.675 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:24.675 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:24.675 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:24.675 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:24.675 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:24.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:24.727 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:24.726+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:24.733 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:24.732+0000 7f9b0e557640 -1 osd.12 107 handle_osd_map injecting map crc failure 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1434: done (10 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1435: done (9 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1436: done (8 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1437: done (7 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1439: done (6 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1440: done (5 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1441: done (4 left) 2024-09-13T13:51:24.736 INFO:tasks.rados.rados.0.smithi159.stdout:1442: done (3 left) 2024-09-13T13:51:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:1443: done (2 left) 2024-09-13T13:51:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:1444: done (1 left) 2024-09-13T13:51:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:1447: done (0 left) 2024-09-13T13:51:24.737 INFO:tasks.rados.rados.0.smithi159.stdout:1448: snap_create 2024-09-13T13:51:24.787 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:24.786+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:24.788 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 96.245073194, 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.789 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.585631128000003, 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:24.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:51:24.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.584242904000007, 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:51:24.798 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:51:24.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.583540901000006, 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:51:24.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:51:24.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:96 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'70, mlcod=25'63)", 2024-09-13T13:51:24.807 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.583055098000003, 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050574289999999996, 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204032+0000", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204033+0000", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204133+0000", 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9450000000000005e-05 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204163+0000", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0018999999999998e-05 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.204168+0000", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5539999999999997e-06 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206261+0000", 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020930889999999998 2024-09-13T13:51:24.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.206274+0000", 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3018e-05 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209065+0000", 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027905909999999998 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209090+0000", 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4896999999999999e-05 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.189938564000002, 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.812 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:116 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'86, mlcod=25'80)", 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.189394125000007, 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051291219999999998, 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597693+0000", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597694+0000", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5900000000000003e-07 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597697+0000", 2024-09-13T13:51:24.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6440000000000002e-06 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597713+0000", 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5699999999999999e-05 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597718+0000", 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1669999999999998e-06 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599712+0000", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0019937420000000002 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599730+0000", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8091e-05 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602796+0000", 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003066306 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602823+0000", 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6112999999999999e-05 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.183814760999994, 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.821 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:51:24.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.825 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.870257605000006, 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:51:24.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:51:24.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:51:24.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:24.832 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.016418857000005, 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:24.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:51:24.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:812 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 2412870~427480,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.597068590000006, 2024-09-13T13:51:24.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054419789999999996, 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 812 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190019+0000", 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190022+0000", 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190034+0000", 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2235000000000001e-05 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190058+0000", 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3323999999999999e-05 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190071+0000", 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3861e-05 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190156+0000", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4179999999999997e-05 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190164+0000", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6589999999999996e-06 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194705+0000", 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045405109999999997 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.194761+0000", 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6140999999999997e-05 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195461+0000", 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00070009399999999996 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.843 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.596983988000005, 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:24.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:51:24.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:51:24.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.596936037999996, 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.852 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:24.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.908473111000006, 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.212865983999997, 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.485163710000002, 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.459654305000001, 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.458988568000002, 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:51:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:51:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:51:24.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 37.458947551999998, 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.884 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:51:24.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.888 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 12.639810644000001, 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:24.889 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:51:24.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:51:24.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.894 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:24.895 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:24.895 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:24.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:24.897 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:24.896+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:24.997 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:24.996+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:25.007 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.007 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:25.098 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:25.096+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:25.123 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.123 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:25.199 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:25.199+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:25.239 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.239 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.248 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:25.300 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:25.299+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:25.333 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:25.355 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:25.355 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:25.397 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.398 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.401 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:25.400+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:25.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:25.502 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:25.501+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:25.554 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.554 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:25.603 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:25.602+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:25.704 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:25.704+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:25.709 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.710 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:25.737 INFO:tasks.rados.rados.0.smithi159.stdout:1448: done (0 left) 2024-09-13T13:51:25.737 INFO:tasks.rados.rados.0.smithi159.stdout:1449: write oid 23 current snap is 32 2024-09-13T13:51:25.737 INFO:tasks.rados.rados.0.smithi159.stdout:1449: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-13T13:51:25.743 INFO:tasks.rados.rados.0.smithi159.stdout:1449: writing smithi15938319-23 from 453855 to 1150842 tid 1 2024-09-13T13:51:25.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:51:25.750 INFO:tasks.rados.rados.0.smithi159.stdout:1449: writing smithi15938319-23 from 1890787 to 2438870 tid 2 2024-09-13T13:51:25.750 INFO:tasks.rados.rados.0.smithi159.stdout:1450: tier_evict oid 1 2024-09-13T13:51:25.750 INFO:tasks.rados.rados.0.smithi159.stdout:1450: tier_evict oid 1 snap -1 2024-09-13T13:51:25.750 INFO:tasks.rados.rados.0.smithi159.stdout:1451: write (excl) oid 18 current snap is 32 2024-09-13T13:51:25.750 INFO:tasks.rados.rados.0.smithi159.stdout:1451: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-13T13:51:25.757 INFO:tasks.rados.rados.0.smithi159.stdout:1451: writing smithi15938319-18 from 470662 to 1242898 tid 1 2024-09-13T13:51:25.762 INFO:tasks.rados.rados.0.smithi159.stdout:1451: writing smithi15938319-18 from 1832916 to 2241010 tid 2 2024-09-13T13:51:25.763 INFO:tasks.rados.rados.0.smithi159.stdout:1452: write (excl) oid 35 current snap is 32 2024-09-13T13:51:25.763 INFO:tasks.rados.rados.0.smithi159.stdout:1449: finishing write tid 1 to smithi15938319-23 2024-09-13T13:51:25.763 INFO:tasks.rados.rados.0.smithi159.stdout:1449: finishing write tid 2 to smithi15938319-23 2024-09-13T13:51:25.763 INFO:tasks.rados.rados.0.smithi159.stdout:1449: finishing write tid 3 to smithi15938319-23 2024-09-13T13:51:25.763 INFO:tasks.rados.rados.0.smithi159.stdout:1452: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-13T13:51:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1452: writing smithi15938319-35 from 487469 to 1051307 tid 1 2024-09-13T13:51:25.775 INFO:tasks.rados.rados.0.smithi159.stdout:1452: writing smithi15938319-35 from 1607751 to 2159503 tid 2 2024-09-13T13:51:25.776 INFO:tasks.rados.rados.0.smithi159.stdout:1453: read oid 7 snap -1 2024-09-13T13:51:25.776 INFO:tasks.rados.rados.0.smithi159.stdout:1453: expect (ObjNum 251 snap 28 seq_num 251) 2024-09-13T13:51:25.776 INFO:tasks.rados.rados.0.smithi159.stdout:1454: rollback oid 16 current snap is 32 2024-09-13T13:51:25.776 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 16 to 21 2024-09-13T13:51:25.776 INFO:tasks.rados.rados.0.smithi159.stdout:1449: finishing write tid 4 to smithi15938319-23 2024-09-13T13:51:25.777 INFO:tasks.rados.rados.0.smithi159.stdout:1449: oid 23 updating version 0 to 192 2024-09-13T13:51:25.777 INFO:tasks.rados.rados.0.smithi159.stdout:1449: oid 23 updating version 192 to 193 2024-09-13T13:51:25.777 INFO:tasks.rados.rados.0.smithi159.stdout:1449: oid 23 version 193 is already newer than 191 2024-09-13T13:51:25.777 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 193 (ObjNum 265 snap 32 seq_num 265) dirty exists 2024-09-13T13:51:25.777 INFO:tasks.rados.rados.0.smithi159.stdout:1449: left oid 23 (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1450: got (22) Invalid argument 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1449: done (5 left) 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1450: done (4 left) 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1455: read oid 5 snap -1 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1455: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:25.778 INFO:tasks.rados.rados.0.smithi159.stdout:1456: snap_create 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: finishing write tid 1 to smithi15938319-18 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: finishing write tid 2 to smithi15938319-18 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: finishing write tid 3 to smithi15938319-18 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: finishing write tid 4 to smithi15938319-18 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: oid 18 updating version 0 to 219 2024-09-13T13:51:25.779 INFO:tasks.rados.rados.0.smithi159.stdout:1451: oid 18 version 219 is already newer than 217 2024-09-13T13:51:25.780 INFO:tasks.rados.rados.0.smithi159.stdout:1451: oid 18 version 219 is already newer than 218 2024-09-13T13:51:25.780 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 219 (ObjNum 266 snap 32 seq_num 266) dirty exists 2024-09-13T13:51:25.780 INFO:tasks.rados.rados.0.smithi159.stdout:1451: left oid 18 (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:51:25.780 INFO:tasks.rados.rados.0.smithi159.stdout:1452: finishing write tid 1 to smithi15938319-35 2024-09-13T13:51:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1452: finishing write tid 2 to smithi15938319-35 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1452: finishing write tid 3 to smithi15938319-35 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1452: finishing write tid 4 to smithi15938319-35 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1452: oid 35 updating version 0 to 256 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1452: oid 35 version 256 is already newer than 254 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1452: oid 35 version 256 is already newer than 255 2024-09-13T13:51:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 256 (ObjNum 267 snap 32 seq_num 267) dirty exists 2024-09-13T13:51:25.783 INFO:tasks.rados.rados.0.smithi159.stdout:1452: left oid 35 (ObjNum 267 snap 32 seq_num 267) 2024-09-13T13:51:25.783 INFO:tasks.rados.rados.0.smithi159.stdout:1453: expect (ObjNum 251 snap 28 seq_num 251) 2024-09-13T13:51:25.784 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:25.806 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:25.804+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:25.832 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.833 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:25.907 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:25.906+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:1454: finishing rollback tid 0 to smithi15938319-16 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:1454: finishing rollback tid 1 to smithi15938319-16 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:1454: finishing rollback tid 2 to smithi15938319-16 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 260 (ObjNum 199 snap 18 seq_num 199) dirty exists 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:1455: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:25.938 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:25.951 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:25.951 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:25.960 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:26.008 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:26.007+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:26.042 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3144542 2024-09-13T13:51:26.069 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.069 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:26.078 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2708917 2024-09-13T13:51:26.109 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:26.108+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:26.186 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.186 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:26.238 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:26.237+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:26.301 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.302 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:26.311 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:26.309+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:26.411 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:26.409+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:26.459 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.459 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:26.512 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:26.511+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:26.583 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.583 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:26.613 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:26.612+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:26.714 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:26.713+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1451: done (5 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1452: done (4 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1453: done (3 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1454: done (2 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1455: done (1 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1456: done (0 left) 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1457: write (excl) oid 50 current snap is 33 2024-09-13T13:51:26.741 INFO:tasks.rados.rados.0.smithi159.stdout:1457: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-13T13:51:26.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:26.746 INFO:tasks.rados.rados.0.smithi159.stdout:1457: writing smithi15938319-50 from 504276 to 1143363 tid 1 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1457: writing smithi15938319-50 from 1666233 to 2361643 tid 2 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1458: read oid 32 snap -1 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1458: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1459: read oid 26 snap -1 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1459: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:26.754 INFO:tasks.rados.rados.0.smithi159.stdout:1460: read oid 9 snap -1 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1460: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1461: read oid 14 snap -1 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1461: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1462: read oid 36 snap -1 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1462: expect deleted 2024-09-13T13:51:26.755 INFO:tasks.rados.rados.0.smithi159.stdout:1463: snap_create 2024-09-13T13:51:26.760 INFO:tasks.rados.rados.0.smithi159.stdout:1459: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:26.789 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:26.791 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:26.789+0000 7faeaa210640 -1 osd.8 109 handle_osd_map injecting map crc failure 2024-09-13T13:51:26.791 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:26.790+0000 7f002dbdb640 -1 osd.15 109 handle_osd_map injecting map crc failure 2024-09-13T13:51:26.815 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:26.813+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:26.849 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.850 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.855 INFO:tasks.rados.rados.0.smithi159.stdout:1461: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:51:26.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:26.865 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1029723 2024-09-13T13:51:26.915 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:26.914+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:26.961 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:26.961 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:26.970 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:26.978 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:26.977+0000 7f9b0c553640 -1 osd.12 109 handle_osd_map injecting map crc failure 2024-09-13T13:51:27.016 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:27.015+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:27.030 INFO:tasks.rados.rados.0.smithi159.stdout:1458: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:27.072 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.073 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.081 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:27.117 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:27.116+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:27.218 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:27.217+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:27.261 INFO:tasks.rados.rados.0.smithi159.stdout:1460: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:27.294 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.294 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:27.318 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 993774 2024-09-13T13:51:27.319 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:27.318+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:27.354 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 501774 2024-09-13T13:51:27.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:27.400 INFO:tasks.rados.rados.0.smithi159.stdout:1457: finishing write tid 1 to smithi15938319-50 2024-09-13T13:51:27.400 INFO:tasks.rados.rados.0.smithi159.stdout:1457: finishing write tid 2 to smithi15938319-50 2024-09-13T13:51:27.400 INFO:tasks.rados.rados.0.smithi159.stdout:1457: finishing write tid 3 to smithi15938319-50 2024-09-13T13:51:27.400 INFO:tasks.rados.rados.0.smithi159.stdout:1457: finishing write tid 4 to smithi15938319-50 2024-09-13T13:51:27.400 INFO:tasks.rados.rados.0.smithi159.stdout:1457: oid 50 updating version 0 to 222 2024-09-13T13:51:27.401 INFO:tasks.rados.rados.0.smithi159.stdout:1457: oid 50 version 222 is already newer than 221 2024-09-13T13:51:27.401 INFO:tasks.rados.rados.0.smithi159.stdout:1457: oid 50 version 222 is already newer than 220 2024-09-13T13:51:27.401 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 222 (ObjNum 268 snap 33 seq_num 268) dirty exists 2024-09-13T13:51:27.401 INFO:tasks.rados.rados.0.smithi159.stdout:1457: left oid 50 (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:27.407 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.407 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:27.458 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:27.457+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:27.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:27.507 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:27.519 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.519 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.520 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:27.518+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:27.527 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:27.529 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:27.530 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:27.552 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:27.552 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:27.573 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:27.594 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:27.621 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:27.620+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:27.670 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.670 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.678 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:27.725 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:27.725+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:27.738 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:27.737+0000 7f002fbdf640 -1 osd.15 110 handle_osd_map injecting map crc failure 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1457: done (6 left) 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1458: done (5 left) 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1459: done (4 left) 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1460: done (3 left) 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1461: done (2 left) 2024-09-13T13:51:27.742 INFO:tasks.rados.rados.0.smithi159.stdout:1462: done (1 left) 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1463: done (0 left) 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1464: read oid 30 snap -1 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1464: expect deleted 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1465: read oid 49 snap 16 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1465: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:27.743 INFO:tasks.rados.rados.0.smithi159.stdout:1466: snap_remove snap 31 2024-09-13T13:51:27.783 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.784 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:27.836 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:27.835+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:27.891 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:27.890+0000 7fe857f4f640 -1 osd.7 110 handle_osd_map injecting map crc failure 2024-09-13T13:51:27.895 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:27.895 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:27.904 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:27.946 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:27.944+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:28.021 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:28.021 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.029 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:28.027+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:28.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:28.130 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:28.128+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:28.176 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.176 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:28.176 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:28.176 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:28.176 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.184 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:28.269 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:28.267+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:28.329 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.429536650000003, 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.330 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:28.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:51:28.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.336 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.413909035000003, 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:51:28.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:28.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:51:28.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:51:28.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:51:28.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.405197360000003, 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:28.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:51:28.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:51:28.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:51:28.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.557651397000001, 2024-09-13T13:51:28.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:28.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:51:28.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:51:28.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:51:28.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:51:28.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:51:28.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:51:28.362 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 59.580911209999996, 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:51:28.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.367 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 58.366240931, 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.368 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.369 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:51:28.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.369 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:51:28.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:51:28.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:51:28.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:51:28.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:51:28.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:51:28.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:51:28.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:51:28.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:51:28.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.379 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.364439304999998, 2024-09-13T13:51:28.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:51:28.381 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.381 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.381 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.381 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.381 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:51:28.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:28.386 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 47.085515737999998, 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:28.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:51:28.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:51:28.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:28.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 47.085492379000002, 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:28.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:51:28.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 36.587047542000001, 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:51:28.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.501426178999999, 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:51:28.407 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:51:28.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:51:28.412 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.566555935, 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:51:28.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:51:28.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:51:28.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2853 4.1 4:8e40678c:::smithi15938319-17:head [sparse-read 0~2202141,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.470671519, 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098372819999999993, 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.419 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2853 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857033+0000", 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3149999999999999e-06 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857041+0000", 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5369999999999997e-06 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857060+0000", 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8972000000000001e-05 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857070+0000", 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3489999999999998e-06 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857145+0000", 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4870000000000007e-05 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857153+0000", 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5310000000000001e-06 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.862925+0000", 2024-09-13T13:51:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057717510000000003 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.863022+0000", 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6618000000000002e-05 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866868+0000", 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038463389999999998 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.424 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.47061162, 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.425 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:51:28.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.430 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.470573759000001, 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:51:28.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:28.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:51:28.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:51:28.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:51:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:236 4.f e95/34 4:f2f09040:::smithi15938319-31:16 v 95'156, mlcod=93'151)", 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.865075+0000", 2024-09-13T13:51:28.437 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.462627266999998, 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087647070000000001, 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865075+0000", 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865075+0000", 2024-09-13T13:51:28.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865076+0000", 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0599999999999999e-07 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865088+0000", 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2361000000000001e-05 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865138+0000", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.865146+0000", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3280000000000006e-06 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866193+0000", 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001046812 2024-09-13T13:51:28.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866213+0000", 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0088000000000001e-05 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873783+0000", 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075693050000000001 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873840+0000", 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7011e-05 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.4.0:124 4.e e95/34 4:721f6cf4:::smithi15938319-18:13 v 95'176, mlcod=94'171)", 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.871591+0000", 2024-09-13T13:51:28.442 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.456111422999999, 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087166489999999999, 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871591+0000", 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871591+0000", 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871591+0000", 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2699999999999999e-07 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871601+0000", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871631+0000", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9788e-05 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.871636+0000", 2024-09-13T13:51:28.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8000000000000004e-06 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872483+0000", 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084681299999999997 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872509+0000", 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5366000000000001e-05 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.880261+0000", 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077522110000000002 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.880307+0000", 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6558999999999999e-05 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.489878888, 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.448 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:51:28.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:51:28.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:28.456 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.489486164000001, 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:28.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:51:28.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:28.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.462 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.489470563999999, 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:51:28.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:51:28.467 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.468 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.468 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.468 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.469 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:28.470 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:28.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:28.472 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:28.468+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:28.472 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:28.486 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:28.508 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:28.508 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:28.530 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:28.530 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:28.552 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:28.553 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:28.570 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:28.569+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:28.595 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:28.595 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:28.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:28.599+0000 7fa5610d8640 0 -- 172.21.15.27:0/1823664445 >> v2:172.21.15.122:6808/1255553857 conn(0x7fa55c221bd0 msgr2=0x7fa55c223fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:28.615 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:28.615 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:28.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:28.624 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:28.671 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:28.671+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:28.726 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.726 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:28.726 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:28.726 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:28.726 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.733 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:28.731+0000 7f1a526ee640 -1 osd.13 111 handle_osd_map injecting map crc failure 2024-09-13T13:51:28.735 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:28.735 INFO:tasks.rados.rados.0.smithi159.stdout:1466: done (2 left) 2024-09-13T13:51:28.736 INFO:tasks.rados.rados.0.smithi159.stdout:1467: read oid 8 snap 32 2024-09-13T13:51:28.736 INFO:tasks.rados.rados.0.smithi159.stdout:1467: expect (ObjNum 261 snap 30 seq_num 261) 2024-09-13T13:51:28.736 INFO:tasks.rados.rados.0.smithi159.stdout:1468: read oid 10 snap 24 2024-09-13T13:51:28.736 INFO:tasks.rados.rados.0.smithi159.stdout:1468: expect deleted 2024-09-13T13:51:28.736 INFO:tasks.rados.rados.0.smithi159.stdout:1469: set_chunk oid 46 target oid 2024-09-13T13:51:28.737 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:28.735+0000 7fe859f53640 -1 osd.7 111 handle_osd_map injecting map crc failure 2024-09-13T13:51:28.747 INFO:tasks.rados.rados.0.smithi159.stdout:1469: set_chunk oid 46 offset: 1654272 length: 463872 target oid 047a89f5b60bdbcedd0db40d6fe8adeeb34fd31d8dadf24af2db5a247da4e00b offset: 0 snap -1 2024-09-13T13:51:28.748 INFO:tasks.rados.rados.0.smithi159.stdout:1470: rollback oid 34 current snap is 34 2024-09-13T13:51:28.748 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 34 to 33 2024-09-13T13:51:28.748 INFO:tasks.rados.rados.0.smithi159.stdout:1465: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:28.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:51:28.753 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1545468 2024-09-13T13:51:28.795 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:28.794+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:28.842 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:28.842 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:28.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.298904921, 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:28.843 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:51:28.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:51:28.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:51:28.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 98.244579032000004, 2024-09-13T13:51:28.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:51:28.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:51:28.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 98.231200642000005, 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:51:28.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 70.417867705000006, 2024-09-13T13:51:28.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:51:28.864 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612455697999998, 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.865 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:28.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.870 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612418525999999, 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.871 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:51:28.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612397798000003, 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:28.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:51:28.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612301496999997, 2024-09-13T13:51:28.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:51:28.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:51:28.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612278852999999, 2024-09-13T13:51:28.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:51:28.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.612252822999999, 2024-09-13T13:51:28.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:51:28.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:28.900 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.472751956000003, 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:28.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:51:28.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.906 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.472645008999997, 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:51:28.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.912 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.472615750000003, 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:28.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.918 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.466251346999996, 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:51:28.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:51:28.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:51:28.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.926 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.461291152999998, 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:51:28.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:51:28.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:51:28.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.519301435999999, 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:51:28.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 53.519262298000001, 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.977832300999999, 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:51:28.946 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:51:28.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.977769355, 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.951 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:28.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:51:28.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.977735673000002, 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:28.957 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:51:28.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:51:28.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:28.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:28.962 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:28.963 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:28.962+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:28.964 INFO:tasks.rados.rados.0.smithi159.stdout:1467: expect (ObjNum 261 snap 30 seq_num 261) 2024-09-13T13:51:28.964 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:28.964 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:28.964 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:28.992 INFO:tasks.rados.rados.0.smithi159.stdout:1469: finishing set_chunk to oid 46 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 96 (ObjNum 247 snap 28 seq_num 247) dirty exists 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1470: finishing rollback tid 0 to smithi15938319-34 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1470: finishing rollback tid 1 to smithi15938319-34 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1464: done (5 left) 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1465: done (4 left) 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1467: done (3 left) 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1468: done (2 left) 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1469: done (1 left) 2024-09-13T13:51:28.993 INFO:tasks.rados.rados.0.smithi159.stdout:1471: read oid 2 snap -1 2024-09-13T13:51:28.994 INFO:tasks.rados.rados.0.smithi159.stdout:1471: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:28.994 INFO:tasks.rados.rados.0.smithi159.stdout:1472: tier_flush oid 12 2024-09-13T13:51:28.994 INFO:tasks.rados.rados.0.smithi159.stdout:1472: tier_flush oid 12 snap -1 2024-09-13T13:51:28.994 INFO:tasks.rados.rados.0.smithi159.stdout:1473: set_chunk oid 50 target oid 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1473: set_chunk oid 50 offset: 471040 length: 2001920 target oid a176621cedf6d45f7ef73c48feb45033e3c341b7b750526a3f95d61651a5515a offset: 0 snap -1 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1474: read oid 28 snap -1 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1474: expect deleted 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1475: write oid 8 current snap is 34 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1472: got (16) Device or resource busy 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 12 v 180 (ObjNum 263 snap 30 seq_num 263) unflushed 30 2024-09-13T13:51:29.015 INFO:tasks.rados.rados.0.smithi159.stdout:1471: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:29.064 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:29.062+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:29.073 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.073 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:29.073 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:29.073 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:29.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:29.144 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:29.165 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:29.164+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:29.214 INFO:tasks.rados.rados.0.smithi159.stdout:1473: finishing set_chunk to oid 50 2024-09-13T13:51:29.215 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 222 (ObjNum 268 snap 33 seq_num 268) dirty exists 2024-09-13T13:51:29.215 INFO:tasks.rados.rados.0.smithi159.stdout:1470: finishing rollback tid 2 to smithi15938319-34 2024-09-13T13:51:29.215 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 162 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-09-13T13:51:29.215 INFO:tasks.rados.rados.0.smithi159.stdout:1475: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-13T13:51:29.220 INFO:tasks.rados.rados.0.smithi159.stdout:1475: writing smithi15938319-8 from 521083 to 1235419 tid 1 2024-09-13T13:51:29.226 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.227 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:29.227 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:29.227 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:29.227 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:29.227 INFO:tasks.rados.rados.0.smithi159.stdout:1475: writing smithi15938319-8 from 1724715 to 2280136 tid 2 2024-09-13T13:51:29.227 INFO:tasks.rados.rados.0.smithi159.stdout:1470: done (5 left) 2024-09-13T13:51:29.227 INFO:tasks.rados.rados.0.smithi159.stdout:1471: done (4 left) 2024-09-13T13:51:29.228 INFO:tasks.rados.rados.0.smithi159.stdout:1472: done (3 left) 2024-09-13T13:51:29.228 INFO:tasks.rados.rados.0.smithi159.stdout:1473: done (2 left) 2024-09-13T13:51:29.228 INFO:tasks.rados.rados.0.smithi159.stdout:1474: done (1 left) 2024-09-13T13:51:29.228 INFO:tasks.rados.rados.0.smithi159.stdout:1476: snap_create 2024-09-13T13:51:29.229 INFO:tasks.rados.rados.0.smithi159.stdout:1475: finishing write tid 1 to smithi15938319-8 2024-09-13T13:51:29.236 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:29.237 INFO:tasks.rados.rados.0.smithi159.stdout:1475: finishing write tid 2 to smithi15938319-8 2024-09-13T13:51:29.239 INFO:tasks.rados.rados.0.smithi159.stdout:1475: finishing write tid 3 to smithi15938319-8 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:1475: finishing write tid 4 to smithi15938319-8 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:1475: oid 8 updating version 0 to 131 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:1475: oid 8 version 131 is already newer than 130 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:1475: oid 8 updating version 131 to 132 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 132 (ObjNum 269 snap 34 seq_num 269) dirty exists 2024-09-13T13:51:29.240 INFO:tasks.rados.rados.0.smithi159.stdout:1475: left oid 8 (ObjNum 269 snap 34 seq_num 269) 2024-09-13T13:51:29.279 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:29.277+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:29.340 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.600972447000004, 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.341 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:51:29.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:51:29.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:51:29.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:51:29.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:51:29.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:51:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:51:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.351 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.351 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:29.351 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.595731825000001, 2024-09-13T13:51:29.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:51:29.352 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.352 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.352 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.352 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.353 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.353 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.353 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:51:29.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.354 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:51:29.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:51:29.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:51:29.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:51:29.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:51:29.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:51:29.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.361 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.595680510999998, 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.362 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:51:29.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:51:29.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:51:29.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:51:29.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.595608314000003, 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:51:29.373 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.374 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:29.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:29.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:51:29.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:51:29.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:51:29.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:51:29.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.385 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.595569308999998, 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.386 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:29.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:51:29.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.392 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.595504896999998, 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:29.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:51:29.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:51:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.570154801000001, 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.399 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:51:29.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:51:29.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.405 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.539593976999996, 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.411 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.518267390000005, 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:51:29.412 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:51:29.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.416 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.440051112999996, 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.418 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:29.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:51:29.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:51:29.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:51:29.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.096079453000002, 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.428 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:51:29.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:51:29.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.096021401000002, 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.436 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:29.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:29.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:29.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:51:29.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.095995442000003, 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.442 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:29.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:51:29.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:51:29.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:51:29.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.095881026000001, 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:51:29.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:51:29.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.458 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.095850134999999, 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:51:29.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:51:29.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:51:29.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:51:29.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:51:29.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:51:29.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:51:29.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:51:29.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.095813552000003, 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:51:29.474 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.475 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:51:29.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:51:29.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.016018682000002, 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.482 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:51:29.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 47.641358263999997, 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:51:29.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:51:29.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.491 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:29.492 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.518166741000002, 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:51:29.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 28.510146443, 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:51:29.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.503 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.503 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.503 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:29.504 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:29.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:29.505 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:29.503+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:29.506 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:29.605 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:29.604+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:29.648 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.648 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:29.648 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:29.648 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:29.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:29.706 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:29.705+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:29.760 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.760 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:29.760 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:29.760 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:29.760 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:29.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:29.810 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:29.809+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:29.909 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:29.908+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:29.912 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.683299082999994, 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.913 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:51:29.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:51:29.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.920 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.178560900999997, 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:51:29.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:51:29.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:51:29.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.927 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.178511588999996, 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:29.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:51:29.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.178495885000004, 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:51:29.934 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:51:29.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.940 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.178468209000002, 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:51:29.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.143250610999999, 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:51:29.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:51:29.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.136181629000006, 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:51:29.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:51:29.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:51:29.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.101643167000006, 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:51:29.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:51:29.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:51:29.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.094013325000006, 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.967 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:51:29.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:51:29.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.973 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.090636953000001, 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:51:29.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:51:29.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.987777672999997, 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:51:29.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 72.776754132999997, 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.986 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:51:29.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:51:29.990 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 72.765238850000003, 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:51:29.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.663930381999997, 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:29.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:29.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:29.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:29.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:51:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.001 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.663866034000002, 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:51:30.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:51:30.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:51:30.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.663837972000003, 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.008 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:51:30.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:51:30.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.662156082000003, 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.014 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:51:30.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:51:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:51:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1811 4.2 4:4d382039:::smithi15938319-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.655103062000002, 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030548815999999999, 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.022 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1811 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255825+0000", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255828+0000", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000002e-06 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255835+0000", 2024-09-13T13:51:30.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2119999999999997e-06 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.255840+0000", 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4079999999999997e-06 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256027+0000", 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018704600000000001 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256075+0000", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260766+0000", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046910119999999996 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260793+0000", 2024-09-13T13:51:30.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6594999999999999e-05 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280369+0000", 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019576296 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280418+0000", 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9221999999999997e-05 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286090+0000", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005671873 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286195+0000", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010464299999999999 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.286374+0000", 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179118 2024-09-13T13:51:30.028 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.222627873999997, 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.029 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:51:30.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.093992118999999, 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:30.033 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:51:30.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.038 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:30.039 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:30.039 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:30.041 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:51:30.042 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1475: done (1 left) 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1476: done (0 left) 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1477: write (excl) oid 12 current snap is 35 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1477: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1477: writing smithi15938319-12 from 537890 to 1327475 tid 1 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1477: writing smithi15938319-12 from 1783197 to 2482276 tid 2 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1478: rollback oid 22 current snap is 35 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 22 to 16 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1479: read oid 13 snap 32 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1479: expect (ObjNum 250 snap 28 seq_num 250) 2024-09-13T13:51:30.044 INFO:tasks.rados.rados.0.smithi159.stdout:1480: snap_create 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: finishing write tid 1 to smithi15938319-12 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: finishing write tid 2 to smithi15938319-12 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1478: finishing rollback tid 0 to smithi15938319-22 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: finishing write tid 3 to smithi15938319-12 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: finishing write tid 4 to smithi15938319-12 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: oid 12 updating version 0 to 194 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: oid 12 updating version 194 to 196 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: oid 12 version 196 is already newer than 195 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 196 (ObjNum 270 snap 35 seq_num 270) dirty exists 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1477: left oid 12 (ObjNum 270 snap 35 seq_num 270) 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1478: finishing rollback tid 1 to smithi15938319-22 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:1479: expect (ObjNum 250 snap 28 seq_num 250) 2024-09-13T13:51:30.045 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:30.046 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 704637 2024-09-13T13:51:30.046 INFO:tasks.rados.rados.0.smithi159.stdout:1478: finishing rollback tid 2 to smithi15938319-22 2024-09-13T13:51:30.046 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 25 (ObjNum 153 snap 12 seq_num 153) dirty exists 2024-09-13T13:51:30.046 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:30.039+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:30.141 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:30.140+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:30.150 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:30.150 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:30.150 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:30.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:30.242 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:30.241+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:30.261 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.261 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:30.261 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:30.261 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:30.261 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:30.269 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:30.343 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:30.342+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:30.415 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.415 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:30.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.681240013999997, 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.416 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:51:30.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:51:30.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:51:30.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.422 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.681162612999998, 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.423 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:51:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.424 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:51:30.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.429 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.681010075000003, 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.430 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:30.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:51:30.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:51:30.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.680904036000001, 2024-09-13T13:51:30.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:30.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:51:30.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:51:30.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.672564957999995, 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.444 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.645198248, 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.450 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:51:30.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:51:30.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.456 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.643431649999997, 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:51:30.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:51:30.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.638552308000001, 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.463 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:51:30.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:51:30.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:219 3.2 e41/33 3:650cb820:::smithi15938319-20 low_tier:head v 41'6, mlcod=41'3)", 2024-09-13T13:51:30.469 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.631461324, 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02650433, 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781964+0000", 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.781965+0000", 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.452e-06 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789891+0000", 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0079261269999999998 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789958+0000", 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6587999999999993e-05 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.789965+0000", 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4569999999999999e-06 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790010+0000", 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4592000000000003e-05 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790030+0000", 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0630000000000001e-05 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808419+0000", 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018388315999999998 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.808468+0000", 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9168000000000001e-05 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.474 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.628040583000001, 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:51:30.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:51:30.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:51:30.480 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.603783293000006, 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.481 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:30.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:51:30.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.600898247999993, 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.487 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:51:30.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:51:30.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.493 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.598694222999995, 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.494 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:51:30.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.591703578999997, 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.500 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:30.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:51:30.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.486711964999998, 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.505 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:51:30.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:51:30.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.663122104000003, 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.511 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:51:30.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.663089313999997, 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:51:30.517 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.547329806999997, 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:51:30.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 34.622727118, 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:51:30.527 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:30.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:30.534 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 6.6725173709999996, 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:51:30.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.539 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:30.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:30.541 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:30.539+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:30.641 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:30.639+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:30.685 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.685 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:30.685 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:30.685 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:30.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:30.742 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:30.741+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:30.796 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.796 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:30.796 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:30.796 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:30.796 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:30.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:30.806 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:30.804+0000 7ffaeece8640 -1 osd.0 112 handle_osd_map injecting map crc failure 2024-09-13T13:51:30.842 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:30.842+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.909 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.166062523999997, 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:51:30.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:51:30.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.913 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:30.914 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.160481265000001, 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.157504653999993, 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.919 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:51:30.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:51:30.923 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.155530483000007, 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.924 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:51:30.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.121771382000006, 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.929 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:51:30.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.933 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.116799959999994, 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:51:30.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:51:30.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.112898669000003, 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:30.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:51:30.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.943 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.072860601000002, 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:30.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:51:30.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.067684759000002, 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.949 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:51:30.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.954 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.991733680999999, 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:51:30.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:51:30.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:51:30.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.986932785999997, 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.961 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:51:30.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 76.832293949000004, 2024-09-13T13:51:30.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:30.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:51:30.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.973 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.709412795999995, 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:30.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.652432462999997, 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:51:30.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.985 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.647119613999998, 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:30.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:51:30.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:51:30.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 49.619341106999997, 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.992 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:51:30.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:51:30.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.110700852000001, 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:30.996 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:30.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:51:30.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 8.9966735920000005, 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:31.001 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:31.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:51:31.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3176 4.6 4:6ce02a65:::smithi15938319-20:head [rollback 12] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.9991077910000001, 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080956770000000008, 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.008 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3176 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908269+0000", 2024-09-13T13:51:31.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908270+0000", 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4499999999999996e-07 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908274+0000", 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7929999999999999e-06 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908282+0000", 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.276e-06 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908286+0000", 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1930000000000003e-06 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908382+0000", 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5829999999999996e-05 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908389+0000", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4219999999999997e-06 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912301+0000", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039123059999999999 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912393+0000", 2024-09-13T13:51:31.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1948999999999997e-05 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.912525+0000", 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013263500000000001 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914618+0000", 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020925269999999998 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916287+0000", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001668803 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916323+0000", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6715000000000003e-05 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.916365+0000", 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1483000000000002e-05 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.014 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 7.9990712330000004, 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:31.015 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:51:31.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:31.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.022 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.025 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1477: done (3 left) 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1478: done (2 left) 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1479: done (1 left) 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1480: done (0 left) 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1481: read oid 34 snap -1 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1481: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:31.026 INFO:tasks.rados.rados.0.smithi159.stdout:1482: set_chunk oid 5 target oid 2024-09-13T13:51:31.027 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:31.024+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:31.126 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:31.125+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:31.168 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:31.169 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:31.169 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.177 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:31.193 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:31.191+0000 7fe0f9e03640 -1 osd.2 113 handle_osd_map injecting map crc failure 2024-09-13T13:51:31.194 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:31.193+0000 7f002dbdb640 -1 osd.15 113 handle_osd_map injecting map crc failure 2024-09-13T13:51:31.226 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:31.225+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:31.281 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.281 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:31.281 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:31.281 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:31.281 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.289 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:31.327 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:31.326+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:31.429 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:31.428+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:31.434 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.171019944999998, 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:51:31.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.440 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.170066446000007, 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.442 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.443 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:51:31.444 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.943816471999995, 2024-09-13T13:51:31.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:31.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.659230186000002, 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:51:31.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.213864405999999, 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.457 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:51:31.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2049 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'52, mlcod=66'48)", 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.109213019000002, 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003216062, 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323542+0000", 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323795+0000", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025235600000000002 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323833+0000", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8479000000000002e-05 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323843+0000", 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0142999999999999e-05 2024-09-13T13:51:31.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323913+0000", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9250000000000003e-05 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.323940+0000", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7047999999999999e-05 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326715+0000", 2024-09-13T13:51:31.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027753679999999998 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326758+0000", 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3248999999999999e-05 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.106319612999997, 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:51:31.467 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.471 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.105570724000003, 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:51:31.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:51:31.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2123 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'60, mlcod=70'56)", 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.693035741999999, 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035624979999999999, 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.739720+0000", 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3299999999999998e-07 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740298+0000", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00057840599999999995 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740767+0000", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00046913499999999997 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740784+0000", 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6487999999999999e-05 2024-09-13T13:51:31.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740877+0000", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2666999999999999e-05 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.740903+0000", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6177999999999999e-05 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743246+0000", 2024-09-13T13:51:31.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023431390000000002 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.743282+0000", 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6152000000000002e-05 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:31.481 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.687236542000001, 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:31.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:51:31.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.682108184000001, 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.486 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:51:31.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:51:31.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.681196628000002, 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.491 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:51:31.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:134 4.8 e78/34 4:1b24d731:::smithi15938319-13:11 v 78'71, mlcod=75'65)", 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.813986866999997, 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033973689999999999, 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618768+0000", 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618769+0000", 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4899999999999995e-07 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618785+0000", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6127999999999999e-05 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618814+0000", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9110000000000001e-05 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618821+0000", 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3130000000000004e-06 2024-09-13T13:51:31.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618874+0000", 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3596e-05 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.618890+0000", 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6203999999999999e-05 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622135+0000", 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032444000000000001 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.622166+0000", 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0969000000000001e-05 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.500 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 34.65311054, 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:51:31.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:138 4.8 e80/34 4:1b24d731:::smithi15938319-13:11 v 80'75, mlcod=78'69)", 2024-09-13T13:51:31.505 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 34.652444215999999, 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038853730000000001, 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780312+0000", 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780322+0000", 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0383999999999999e-05 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780367+0000", 2024-09-13T13:51:31.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4783000000000001e-05 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780378+0000", 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0292000000000001e-05 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780701+0000", 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032313399999999999 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780722+0000", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1353e-05 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784153+0000", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003430986 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784196+0000", 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3498999999999998e-05 2024-09-13T13:51:31.510 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:144 4.8 e84/34 4:165e03eb:::smithi15938319-32:a v 84'77, mlcod=80'73)", 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 30.633119022999999, 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032387330000000001, 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799636+0000", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799637+0000", 2024-09-13T13:51:31.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9299999999999999e-07 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.799641+0000", 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2629999999999997e-06 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800327+0000", 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00068558599999999996 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800335+0000", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800408+0000", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3792000000000001e-05 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.800431+0000", 2024-09-13T13:51:31.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2883000000000001e-05 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.802832+0000", 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0024011869999999999 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:00.802875+0000", 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2663999999999997e-05 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2559 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'82, mlcod=84'79)", 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.587210245000001, 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040899279999999996, 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:31.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845546+0000", 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845704+0000", 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015839000000000001 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845732+0000", 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7736e-05 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845738+0000", 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845791+0000", 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845807+0000", 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5755000000000001e-05 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849592+0000", 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037853499999999998 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849635+0000", 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2669000000000001e-05 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.583836196, 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:51:31.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:31.526 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.583068147999999, 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:51:31.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.531 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 24.226754868, 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:51:31.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.536 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.537 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.537 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:31.539 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:31.537+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:31.539 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:31.639 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:31.639+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:31.684 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:31.684 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:31.685 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.685 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:31.685 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:31.685 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.693 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:31.707 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:31.707 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:31.714 INFO:tasks.rados.rados.0.smithi159.stdout:1482: set_chunk oid 5 offset: 1681408 length: 688128 target oid ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86 offset: 0 snap 30 2024-09-13T13:51:31.714 INFO:tasks.rados.rados.0.smithi159.stdout:1483: read oid 2 snap -1 2024-09-13T13:51:31.715 INFO:tasks.rados.rados.0.smithi159.stdout:1483: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:31.715 INFO:tasks.rados.rados.0.smithi159.stdout:1484: read oid 32 snap -1 2024-09-13T13:51:31.715 INFO:tasks.rados.rados.0.smithi159.stdout:1484: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:31.715 INFO:tasks.rados.rados.0.smithi159.stdout:1485: write (excl) oid 20 current snap is 36 2024-09-13T13:51:31.715 INFO:tasks.rados.rados.0.smithi159.stdout:1485: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-13T13:51:31.720 INFO:tasks.rados.rados.0.smithi159.stdout:1485: writing smithi15938319-20 from 554697 to 1019531 tid 1 2024-09-13T13:51:31.727 INFO:tasks.rados.rados.0.smithi159.stdout:1485: writing smithi15938319-20 from 1725326 to 2284416 tid 2 2024-09-13T13:51:31.727 INFO:tasks.rados.rados.0.smithi159.stdout:1486: set_chunk oid 40 target oid 2024-09-13T13:51:31.729 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:31.738 INFO:tasks.rados.rados.0.smithi159.stdout:1486: set_chunk oid 40 offset: 157184 length: 856064 target oid ac0791a1375f052e763a27ad391c3363cd55755906ff9cacb4293d9f87f40c7a offset: 0 snap -1 2024-09-13T13:51:31.739 INFO:tasks.rados.rados.0.smithi159.stdout:1481: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:31.739 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 249340 2024-09-13T13:51:31.740 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:31.739+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:31.750 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:31.750 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:31.771 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:31.771 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:31.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:31.793 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:31.822 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:31.822 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:31.823 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:31.823 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:31.827 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1570736 2024-09-13T13:51:31.831 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:31.840 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:31.839+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:31.881 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:31.912 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:31.912 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:31.934 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:31.934 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:31.941 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:31.940+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:31.954 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:31.955 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1482: finishing set_chunk to oid 5 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1487: read oid 3 snap 28 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1487: expect (ObjNum 239 snap 26 seq_num 239) 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1481: done (6 left) 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1482: done (5 left) 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1488: write oid 11 current snap is 36 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:1484: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:31.960 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:31.976 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:31.976 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.436415994, 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:51:31.981 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.985 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.776973928, 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:51:31.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.990 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.77558570399999, 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:51:31.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:51:31.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'63)", 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.77488370099999, 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055171040000000001, 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:31.995 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203547+0000", 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203548+0000", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2699999999999998e-07 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203554+0000", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.246e-06 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203585+0000", 2024-09-13T13:51:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1118999999999998e-05 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.203592+0000", 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205979+0000", 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023873179999999998 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:31.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205993+0000", 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3828999999999999e-05 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209040+0000", 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030474030000000002 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209064+0000", 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3612e-05 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:31.999 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:115 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'85, mlcod=25'80)", 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.381281364, 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055252610000000001, 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597149+0000", 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.65e-07 2024-09-13T13:51:32.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597155+0000", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8540000000000002e-06 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597173+0000", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7501999999999999e-05 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.597178+0000", 2024-09-13T13:51:32.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.536e-06 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599435+0000", 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0022571729999999999 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.599449+0000", 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3071000000000001e-05 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602647+0000", 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031984380000000001 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.602674+0000", 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7322e-05 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.004 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.37515756099999, 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:51:32.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.061600404999993, 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.009 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:51:32.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:51:32.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 78.207761657000006, 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.015 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:51:32.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:51:32.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 73.788326788000006, 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.021 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:51:32.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.026 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 73.788278837999997, 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:51:32.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:51:32.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.099815910999993, 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:51:32.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:51:32.038 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 46.404208783999998, 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.039 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:32.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:51:32.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.676506510000003, 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.045 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:32.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:51:32.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.650997105000002, 2024-09-13T13:51:32.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:51:32.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:51:32.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.650331368000003, 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.059 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 44.650290351999999, 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:51:32.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.064 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.831153444000002, 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:51:32.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3275 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:32.070 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.0361743510000001, 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061117630000000001, 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3275 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942258+0000", 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9369999999999998e-06 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942265+0000", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-06 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942282+0000", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6880000000000001e-05 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942292+0000", 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8689999999999999e-06 2024-09-13T13:51:32.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7336000000000005e-05 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942388+0000", 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3090000000000001e-06 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945082+0000", 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026939160000000002 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945163+0000", 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0728000000000004e-05 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945384+0000", 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221375 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948177+0000", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027923119999999999 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948231+0000", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948299+0000", 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8300000000000007e-05 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948368+0000", 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8872000000000001e-05 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3276 4.9 4:977d569f:::smithi15938319-22:head [rollback 12] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.0361098659999999, 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082025069999999995, 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.077 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3276 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942325+0000", 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.512e-06 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942335+0000", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0016e-05 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942341+0000", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942471+0000", 2024-09-13T13:51:32.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013015300000000001 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942478+0000", 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.55e-06 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946541+0000", 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040631779999999998 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946640+0000", 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8694000000000001e-05 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946923+0000", 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028351899999999998 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948388+0000", 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001464414 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950419+0000", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020313689999999999 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950465+0000", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5519999999999998e-05 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950523+0000", 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.842e-05 2024-09-13T13:51:32.083 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3277 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 2.0360515449999999, 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010801006, 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:32.084 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3277 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942380+0000", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942384+0000", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1949999999999996e-06 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942391+0000", 2024-09-13T13:51:32.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0339999999999999e-06 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942396+0000", 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8409999999999999e-06 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942508+0000", 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112458 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942512+0000", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.579e-06 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947640+0000", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051287859999999998 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947737+0000", 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6329999999999994e-05 2024-09-13T13:51:32.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947934+0000", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019687900000000001 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948478+0000", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000544448 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953066+0000", 2024-09-13T13:51:32.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00458831 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953129+0000", 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2280999999999995e-05 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953180+0000", 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.090 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:32.091 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:32.091 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:32.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:51:32.092 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:32.093 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:32.091+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:32.093 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:32.093 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:32.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:32.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:32.095 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:32.193 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:32.192+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:32.203 INFO:tasks.rados.rados.0.smithi159.stdout:1483: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:32.205 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.205 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:32.299 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:32.298+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:32.363 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.363 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.372 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:32.407 INFO:tasks.rados.rados.0.smithi159.stdout:1485: finishing write tid 1 to smithi15938319-20 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: finishing write tid 2 to smithi15938319-20 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: finishing write tid 3 to smithi15938319-20 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: finishing write tid 4 to smithi15938319-20 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: oid 20 updating version 0 to 109 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: oid 20 updating version 109 to 110 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: oid 20 updating version 110 to 111 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 111 (ObjNum 271 snap 36 seq_num 271) dirty exists 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1485: left oid 20 (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:51:32.408 INFO:tasks.rados.rados.0.smithi159.stdout:1486: finishing set_chunk to oid 40 2024-09-13T13:51:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 40 set_chunk returned error code -95 offset: 157184 length: 856064 2024-09-13T13:51:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 0 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:51:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1488: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-13T13:51:32.413 INFO:tasks.rados.rados.0.smithi159.stdout:1488: writing smithi15938319-11 from 571504 to 1111587 tid 1 2024-09-13T13:51:32.414 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:32.412+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1488: writing smithi15938319-11 from 1783808 to 2486556 tid 2 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1483: done (5 left) 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1484: done (4 left) 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1485: done (3 left) 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1486: done (2 left) 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1489: snap_create 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:1487: expect (ObjNum 239 snap 26 seq_num 239) 2024-09-13T13:51:32.421 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:32.422 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:32.422 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:32.443 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:32.496 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:32.495+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:32.513 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1776675 2024-09-13T13:51:32.521 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.521 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.523 INFO:tasks.rados.rados.0.smithi159.stdout:1488: finishing write tid 1 to smithi15938319-11 2024-09-13T13:51:32.523 INFO:tasks.rados.rados.0.smithi159.stdout:1488: finishing write tid 2 to smithi15938319-11 2024-09-13T13:51:32.523 INFO:tasks.rados.rados.0.smithi159.stdout:1488: finishing write tid 3 to smithi15938319-11 2024-09-13T13:51:32.523 INFO:tasks.rados.rados.0.smithi159.stdout:1488: finishing write tid 4 to smithi15938319-11 2024-09-13T13:51:32.523 INFO:tasks.rados.rados.0.smithi159.stdout:1488: oid 11 updating version 0 to 191 2024-09-13T13:51:32.524 INFO:tasks.rados.rados.0.smithi159.stdout:1488: oid 11 updating version 191 to 192 2024-09-13T13:51:32.524 INFO:tasks.rados.rados.0.smithi159.stdout:1488: oid 11 version 192 is already newer than 190 2024-09-13T13:51:32.524 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 192 (ObjNum 272 snap 36 seq_num 272) dirty exists 2024-09-13T13:51:32.524 INFO:tasks.rados.rados.0.smithi159.stdout:1488: left oid 11 (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:51:32.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:32.596 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:32.595+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:32.637 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.637 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:32.698 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:32.697+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:32.752 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.752 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:32.799 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:32.798+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:32.868 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.868 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:32.900 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:32.899+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:32.942 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:51:32.948 INFO:tasks.rados.rados.0.smithi159.stdout:1487: done (2 left) 2024-09-13T13:51:32.948 INFO:tasks.rados.rados.0.smithi159.stdout:1488: done (1 left) 2024-09-13T13:51:32.948 INFO:tasks.rados.rados.0.smithi159.stdout:1489: done (0 left) 2024-09-13T13:51:32.948 INFO:tasks.rados.rados.0.smithi159.stdout:1490: write oid 13 current snap is 37 2024-09-13T13:51:32.948 INFO:tasks.rados.rados.0.smithi159.stdout:1490: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-13T13:51:32.954 INFO:tasks.rados.rados.0.smithi159.stdout:1490: writing smithi15938319-13 from 588311 to 1203643 tid 1 2024-09-13T13:51:32.961 INFO:tasks.rados.rados.0.smithi159.stdout:1490: writing smithi15938319-13 from 1842290 to 2405049 tid 2 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1491: set_chunk oid 33 target oid 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1491: set_chunk oid 33 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1492: read oid 1 snap -1 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1492: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1493: write oid 6 current snap is 37 2024-09-13T13:51:32.962 INFO:tasks.rados.rados.0.smithi159.stdout:1493: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-13T13:51:32.967 INFO:tasks.rados.rados.0.smithi159.stdout:1493: writing smithi15938319-6 from 605118 to 1012052 tid 1 2024-09-13T13:51:32.975 INFO:tasks.rados.rados.0.smithi159.stdout:1493: writing smithi15938319-6 from 1617125 to 2323542 tid 2 2024-09-13T13:51:32.975 INFO:tasks.rados.rados.0.smithi159.stdout:1494: snap_remove snap 26 2024-09-13T13:51:32.987 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:32.987 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:32.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:33.001 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:33.000+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:33.102 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:33.101+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:33.143 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.143 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:33.204 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:33.202+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:33.259 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.259 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:33.311 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:33.310+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:33.375 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.375 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:33.406 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:33.405+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:33.442 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:33.440+0000 7f3c35934640 -1 osd.5 114 handle_osd_map injecting map crc failure 2024-09-13T13:51:33.506 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:33.505+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.530 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:33.608 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:33.606+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:33.645 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.646 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.654 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:33.708 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:33.707+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:33.742 INFO:tasks.rados.rados.0.smithi159.stdout:1494: done (4 left) 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1495: delete oid 38 current snap is 37 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: finishing write tid 1 to smithi15938319-13 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1491: finishing set_chunk to oid 33 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1491: got expected ERANGE (src dne) 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: finishing write tid 2 to smithi15938319-13 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: finishing write tid 3 to smithi15938319-13 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: finishing write tid 4 to smithi15938319-13 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: oid 13 updating version 0 to 118 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: oid 13 updating version 118 to 120 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: oid 13 version 120 is already newer than 119 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 120 (ObjNum 273 snap 37 seq_num 273) dirty exists 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1490: left oid 13 (ObjNum 273 snap 37 seq_num 273) 2024-09-13T13:51:33.743 INFO:tasks.rados.rados.0.smithi159.stdout:1492: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:33.762 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:33.773 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.773 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:33.796 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:33.809 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:33.808+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:33.829 INFO:tasks.rados.rados.0.smithi159.stdout:1493: finishing write tid 1 to smithi15938319-6 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: finishing write tid 2 to smithi15938319-6 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: finishing write tid 3 to smithi15938319-6 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: finishing write tid 4 to smithi15938319-6 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: oid 6 updating version 0 to 44 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: oid 6 updating version 44 to 45 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: oid 6 updating version 45 to 46 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 46 (ObjNum 274 snap 37 seq_num 274) dirty exists 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: left oid 6 (ObjNum 274 snap 37 seq_num 274) 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1490: done (4 left) 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1491: done (3 left) 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1492: done (2 left) 2024-09-13T13:51:33.830 INFO:tasks.rados.rados.0.smithi159.stdout:1493: done (1 left) 2024-09-13T13:51:33.831 INFO:tasks.rados.rados.0.smithi159.stdout:1495: done (0 left) 2024-09-13T13:51:33.831 INFO:tasks.rados.rados.0.smithi159.stdout:1496: read oid 17 snap -1 2024-09-13T13:51:33.831 INFO:tasks.rados.rados.0.smithi159.stdout:1496: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:33.831 INFO:tasks.rados.rados.0.smithi159.stdout:1497: set_chunk oid 41 target oid 2024-09-13T13:51:33.858 INFO:tasks.rados.rados.0.smithi159.stdout:1497: set_chunk oid 41 offset: 573952 length: 1517568 target oid 74c77b43c4a6f80bd0e501c2c0488831026fe51f029b77bc514d560d87219b3a offset: 0 snap -1 2024-09-13T13:51:33.858 INFO:tasks.rados.rados.0.smithi159.stdout:1498: set_chunk oid 6 target oid 2024-09-13T13:51:33.869 INFO:tasks.rados.rados.0.smithi159.stdout:1498: set_chunk oid 6 offset: 85504 length: 1233408 target oid a35c5b2ef33c53bd38a6cd00b3a182e7159e6eee00c7c00de0557c4db7ae7174 offset: 0 snap 34 2024-09-13T13:51:33.869 INFO:tasks.rados.rados.0.smithi159.stdout:1499: read oid 43 snap -1 2024-09-13T13:51:33.869 INFO:tasks.rados.rados.0.smithi159.stdout:1499: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:33.870 INFO:tasks.rados.rados.0.smithi159.stdout:1500: delete oid 15 current snap is 37 2024-09-13T13:51:33.870 INFO:tasks.rados.rados.0.smithi159.stdout:1496: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:33.886 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.886 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:33.895 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:33.898 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2314158 2024-09-13T13:51:33.909 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:33.909+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:33.998 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:33.998 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:34.010 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:34.009+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:34.030 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2698957 2024-09-13T13:51:34.053 INFO:tasks.rados.rados.0.smithi159.stdout:1497: finishing set_chunk to oid 41 2024-09-13T13:51:34.053 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 174 (ObjNum 255 snap 30 seq_num 255) dirty exists 2024-09-13T13:51:34.053 INFO:tasks.rados.rados.0.smithi159.stdout:1498: finishing set_chunk to oid 6 2024-09-13T13:51:34.053 INFO:tasks.rados.rados.0.smithi159.stdout:1499: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:34.111 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:34.110+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:34.112 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.112 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.120 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:34.193 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1126092 2024-09-13T13:51:34.212 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:34.211+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:34.225 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.225 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:34.313 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:34.312+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:34.336 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.336 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:34.414 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:34.413+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:34.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:34.487 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.487 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.496 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:34.517 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:34.515+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:34.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:34.592 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:34.598 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.598 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1496: done (4 left) 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1497: done (3 left) 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1498: done (2 left) 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1499: done (1 left) 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1500: done (0 left) 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1501: read oid 1 snap -1 2024-09-13T13:51:34.608 INFO:tasks.rados.rados.0.smithi159.stdout:1501: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:34.609 INFO:tasks.rados.rados.0.smithi159.stdout:1502: read oid 15 snap 21 2024-09-13T13:51:34.609 INFO:tasks.rados.rados.0.smithi159.stdout:1502: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:34.609 INFO:tasks.rados.rados.0.smithi159.stdout:1503: set_chunk oid 4 target oid 2024-09-13T13:51:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:34.614 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:34.622 INFO:tasks.rados.rados.0.smithi159.stdout:1503: set_chunk oid 4 offset: 667136 length: 1079808 target oid 4ee581ad890369ee3aa0fb236d6eafcab44878457e4f5fa4c24768bc20594d1b offset: 0 snap -1 2024-09-13T13:51:34.622 INFO:tasks.rados.rados.0.smithi159.stdout:1504: set_chunk oid 43 target oid 2024-09-13T13:51:34.629 INFO:tasks.rados.rados.0.smithi159.stdout:1504: set_chunk oid 43 offset: 458240 length: 650752 target oid f72d42a949328e2bf278c731ca86a81df414f091bc9ab4aaeac3b2d87e600ad3 offset: 0 snap 21 2024-09-13T13:51:34.629 INFO:tasks.rados.rados.0.smithi159.stdout:1505: snap_create 2024-09-13T13:51:34.629 INFO:tasks.rados.rados.0.smithi159.stdout:1501: expect (ObjNum 176 snap 16 seq_num 176) 2024-09-13T13:51:34.637 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:34.653 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:34.652+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:34.658 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:34.659 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:34.679 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:34.679 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:34.700 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:34.711 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.711 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.717 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:34.716+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:34.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:34.722 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:34.723 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:34.727 INFO:tasks.rados.rados.0.smithi159.stdout:1502: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:51:34.727 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:34.743 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:34.744 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:34.764 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:34.764 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:34.785 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:34.785 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:34.796 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1515675 2024-09-13T13:51:34.806 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:34.818 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:34.817+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:34.827 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:34.828 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:34.848 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:34.848 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:34.864 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:34.864 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:34.870 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:34.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:34.875 INFO:tasks.rados.rados.0.smithi159.stdout:1503: finishing set_chunk to oid 4 2024-09-13T13:51:34.875 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 151 (ObjNum 214 snap 23 seq_num 214) dirty exists 2024-09-13T13:51:34.876 INFO:tasks.rados.rados.0.smithi159.stdout:1504: finishing set_chunk to oid 43 2024-09-13T13:51:34.892 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:34.892 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:34.891+0000 7f1cf4863640 0 -- 172.21.15.27:0/1262623621 >> v2:172.21.15.159:6805/4073091144 conn(0x7f1cf0223140 msgr2=0x7f1cf0225530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:34.919 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:34.918+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:34.953 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:34.952+0000 7faeac214640 -1 osd.8 116 handle_osd_map injecting map crc failure 2024-09-13T13:51:34.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1501: done (4 left) 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1502: done (3 left) 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1503: done (2 left) 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1504: done (1 left) 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1505: done (0 left) 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1506: write oid 22 current snap is 38 2024-09-13T13:51:34.958 INFO:tasks.rados.rados.0.smithi159.stdout:1506: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-13T13:51:34.964 INFO:tasks.rados.rados.0.smithi159.stdout:1506: writing smithi15938319-22 from 655539 to 1288220 tid 1 2024-09-13T13:51:34.965 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:34.964+0000 7f002fbdf640 -1 osd.15 116 handle_osd_map injecting map crc failure 2024-09-13T13:51:34.972 INFO:tasks.rados.rados.0.smithi159.stdout:1506: writing smithi15938319-22 from 2076218 to 2646315 tid 2 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1507: read oid 26 snap -1 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1507: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1508: read oid 32 snap -1 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1508: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1509: read oid 3 snap 36 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1509: expect (ObjNum 254 snap 30 seq_num 254) 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1510: read oid 36 snap -1 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1510: expect deleted 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1511: read oid 17 snap -1 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1511: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1512: read oid 34 snap -1 2024-09-13T13:51:34.973 INFO:tasks.rados.rados.0.smithi159.stdout:1512: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:34.974 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:34.971+0000 7f4ffdd7e640 -1 osd.1 116 handle_osd_map injecting map crc failure 2024-09-13T13:51:34.974 INFO:tasks.rados.rados.0.smithi159.stdout:1513: tier_flush oid 31 2024-09-13T13:51:34.974 INFO:tasks.rados.rados.0.smithi159.stdout:1513: tier_flush oid 31 snap -1 2024-09-13T13:51:34.974 INFO:tasks.rados.rados.0.smithi159.stdout:1514: set_chunk oid 1 target oid 2024-09-13T13:51:34.977 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:34.977 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:34.977 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:34.977 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:34.977 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:34.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:34.989 INFO:tasks.rados.rados.0.smithi159.stdout:1514: set_chunk oid 1 offset: 636928 length: 1289728 target oid a7aa59e690ee297c5113c6761de480eb7f74f9373ff51270cb56954c60da5af4 offset: 0 snap -1 2024-09-13T13:51:34.990 INFO:tasks.rados.rados.0.smithi159.stdout:1515: set_chunk oid 30 target oid 2024-09-13T13:51:34.990 INFO:tasks.rados.rados.0.smithi159.stdout:1515: set_chunk oid 30 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:34.990 INFO:tasks.rados.rados.0.smithi159.stdout:1516: set_chunk oid 11 target oid 2024-09-13T13:51:34.993 INFO:tasks.rados.rados.0.smithi159.stdout:1516: set_chunk oid 11 offset: 1052672 length: 229376 target oid 0b5ccfe58cdfa2904883c0fd0ee5915ad5800b5117ef9a8a4b71f9c84fc9a75b offset: 0 snap 37 2024-09-13T13:51:34.994 INFO:tasks.rados.rados.0.smithi159.stdout:1517: write (excl) oid 33 current snap is 38 2024-09-13T13:51:34.994 INFO:tasks.rados.rados.0.smithi159.stdout:1517: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-13T13:51:35.000 INFO:tasks.rados.rados.0.smithi159.stdout:1517: writing smithi15938319-33 from 672346 to 1380276 tid 1 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1517: writing smithi15938319-33 from 2134700 to 2672346 tid 2 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1518: read oid 6 snap 34 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1518: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1519: tier_promote oid 47 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1506: finishing write tid 1 to smithi15938319-22 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1513: got (16) Device or resource busy 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 31 v 174 (ObjNum 138 snap 11 seq_num 138) unflushed 30 2024-09-13T13:51:35.009 INFO:tasks.rados.rados.0.smithi159.stdout:1506: finishing write tid 2 to smithi15938319-22 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1506: finishing write tid 3 to smithi15938319-22 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1513: done (13 left) 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1520: read oid 45 snap -1 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1520: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1521: read oid 49 snap -1 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1521: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1522: read oid 41 snap -1 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1522: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1506: finishing write tid 4 to smithi15938319-22 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1506: oid 22 updating version 0 to 31 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1506: oid 22 version 31 is already newer than 30 2024-09-13T13:51:35.010 INFO:tasks.rados.rados.0.smithi159.stdout:1506: oid 22 version 31 is already newer than 29 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 31 (ObjNum 277 snap 38 seq_num 277) dirty exists 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1506: left oid 22 (ObjNum 277 snap 38 seq_num 277) 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1506: done (15 left) 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1510: done (14 left) 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1523: read oid 29 snap -1 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1523: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1524: delete oid 35 current snap is 38 2024-09-13T13:51:35.011 INFO:tasks.rados.rados.0.smithi159.stdout:1507: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:35.018 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 868502 2024-09-13T13:51:35.019 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:35.018+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.092 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.323456718000003, 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:51:35.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:51:35.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.098 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.322479520000002, 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:51:35.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:51:35.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:51:35.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.320724854999995, 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:51:35.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:51:35.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:35.111 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.320332641999997, 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:51:35.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.320295314999996, 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:51:35.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:51:35.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:51:35.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.315496961999997, 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:51:35.124 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:51:35.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.130 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.299861737000001, 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.131 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:51:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:51:35.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.299755387000005, 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:51:35.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.291821616999997, 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.143 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:51:35.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:35.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.273558872999999, 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:51:35.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:51:35.155 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.273433058999998, 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:35.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:51:35.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.273306269000003, 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:51:35.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.192107845999999, 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:51:35.168 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:51:35.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.174 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.192006395000007, 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:51:35.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 70.630653628999994, 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:51:35.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:51:35.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:51:35.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:51:35.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 70.624802086000003, 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.187 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:51:35.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:51:35.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:35.191 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 66.347992450000007, 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:51:35.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:51:35.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:35.196 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 61.362924634999999, 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:51:35.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 60.356903359999997, 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.205 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:51:35.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2394 3.1 3:93ff5947:::3634ac6462bf9cb96fe06c81293bb3c6c0e95892f99c8f55c0a14944e78c533e:head [write 0~1120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.311515503000003, 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012999503000000001, 2024-09-13T13:51:35.211 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2394 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778747+0000", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.886e-06 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780479+0000", 2024-09-13T13:51:35.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017320300000000001 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780496+0000", 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7070000000000001e-05 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780522+0000", 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6588999999999999e-05 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780613+0000", 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0382000000000003e-05 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780619+0000", 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.353e-06 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786863+0000", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062442749999999997 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786980+0000", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116236 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.787126+0000", 2024-09-13T13:51:35.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014658299999999999 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791421+0000", 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042944840000000003 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791672+0000", 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025098399999999999 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791713+0000", 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1306000000000001e-05 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791744+0000", 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1325000000000002e-05 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:35.218 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:35.220 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:35.220 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:35.219+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:35.221 INFO:tasks.rados.rados.0.smithi159.stdout:1508: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:51:35.221 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1237202 2024-09-13T13:51:35.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:35.321 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:35.320+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:35.332 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:35.333 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:35.333 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:35.333 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:35.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:35.363 INFO:tasks.rados.rados.0.smithi159.stdout:1512: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:35.413 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3604685 2024-09-13T13:51:35.419 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:35.422 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:35.421+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:35.445 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:35.445 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:35.445 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:35.445 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:35.445 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:35.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:35.523 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:35.523+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.658154163000006, 2024-09-13T13:51:35.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:51:35.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.642526548000006, 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.565 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:51:35.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:51:35.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.633814873000006, 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:51:35.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.576 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.786268910000004, 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.577 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:51:35.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.582 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 66.809528723, 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:35.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:51:35.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.594858443999996, 2024-09-13T13:51:35.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:35.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:51:35.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 62.593056818000001, 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.593 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:51:35.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.314133251000001, 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.597 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:51:35.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:51:35.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:35.602 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.314109891999998, 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:51:35.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.815665054999997, 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:51:35.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:51:35.614 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 34.730043692000002, 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:51:35.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.795173448, 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.619 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:51:35.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2853 4.1 4:8e40678c:::smithi15938319-17:head [sparse-read 0~2202141,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.699289031999999, 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098372819999999993, 2024-09-13T13:51:35.625 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2853 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857033+0000", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3149999999999999e-06 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857041+0000", 2024-09-13T13:51:35.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5369999999999997e-06 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857060+0000", 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8972000000000001e-05 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857070+0000", 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3489999999999998e-06 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857145+0000", 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4870000000000007e-05 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857153+0000", 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5310000000000001e-06 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.862925+0000", 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057717510000000003 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.863022+0000", 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6618000000000002e-05 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866868+0000", 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038463389999999998 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.699229132999999, 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:35.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:51:35.635 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.699191272, 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.636 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:51:35.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:51:35.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.641 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.718496400999999, 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.642 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:51:35.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.718103676999998, 2024-09-13T13:51:35.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:51:35.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:51:35.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 22.718088077000001, 2024-09-13T13:51:35.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:51:35.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:51:35.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.659 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3367 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.58331871199999996, 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096522090000000001, 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.660 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3367 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973002+0000", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2799999999999995e-07 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973007+0000", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9799999999999998e-06 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973015+0000", 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973023+0000", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973140+0000", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011665100000000001 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973146+0000", 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4960000000000001e-06 2024-09-13T13:51:35.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980078+0000", 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069321570000000004 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980132+0000", 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3834e-05 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.982653+0000", 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025209910000000002 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3399 4.1 4:80c58c2d:::smithi15938319-35:head [delete] snapc 27=[27,26,25,24,23,22,20,1e,1b,1a,17,14,12] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.54773818100000005, 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088886150000000008, 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3399 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:35.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008582+0000", 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4300000000000003e-07 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008584+0000", 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3410000000000001e-06 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008591+0000", 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4320000000000004e-06 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008598+0000", 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:35.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015266+0000", 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066677849999999999 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015323+0000", 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7108000000000003e-05 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015457+0000", 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013399700000000001 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017064+0000", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016070769999999999 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017414+0000", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00034987400000000002 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017445+0000", 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0917999999999998e-05 2024-09-13T13:51:35.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017470+0000", 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5576e-05 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:35.671 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:35.673 INFO:tasks.rados.rados.0.smithi159.stdout:1509: expect (ObjNum 254 snap 30 seq_num 254) 2024-09-13T13:51:35.673 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1270396 2024-09-13T13:51:35.673 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:35.673 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:35.674 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:35.672+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:35.676 INFO:tasks.rados.rados.0.smithi159.stdout:1515: finishing set_chunk to oid 30 2024-09-13T13:51:35.676 INFO:tasks.rados.rados.0.smithi159.stdout:1515: got expected ERANGE (src dne) 2024-09-13T13:51:35.676 INFO:tasks.rados.rados.0.smithi159.stdout:1511: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:35.697 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1702967 2024-09-13T13:51:35.774 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:35.773+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:35.777 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:35.777 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:35.777 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:35.777 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:35.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:35.789 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1514: finishing set_chunk to oid 1 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 86 (ObjNum 176 snap 16 seq_num 176) dirty exists 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1516: finishing set_chunk to oid 11 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1519: got (0) Success 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 144 (ObjNum 262 snap 30 seq_num 262) dirty exists 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1517: finishing write tid 1 to smithi15938319-33 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1507: done (15 left) 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1508: done (14 left) 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1509: done (13 left) 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1511: done (12 left) 2024-09-13T13:51:35.860 INFO:tasks.rados.rados.0.smithi159.stdout:1512: done (11 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1514: done (10 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1515: done (9 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1516: done (8 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1519: done (7 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1524: done (6 left) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1525: read oid 43 snap -1 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1525: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1526: read oid 34 snap -1 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1526: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1527: write (excl) oid 36 current snap is 38 2024-09-13T13:51:35.861 INFO:tasks.rados.rados.0.smithi159.stdout:1527: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-13T13:51:35.865 INFO:tasks.rados.rados.0.smithi159.stdout:1527: writing smithi15938319-36 from 705960 to 1164388 tid 1 2024-09-13T13:51:35.873 INFO:tasks.rados.rados.0.smithi159.stdout:1527: writing smithi15938319-36 from 1735311 to 2452735 tid 2 2024-09-13T13:51:35.873 INFO:tasks.rados.rados.0.smithi159.stdout:1528: set_chunk oid 3 target oid 2024-09-13T13:51:35.875 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:35.874+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:35.881 INFO:tasks.rados.rados.0.smithi159.stdout:1528: set_chunk oid 3 offset: 103424 length: 782336 target oid c6a59e045c2a6f777569be441f9851d4a677480fb8299b4c71c52f2e43cc0e62 offset: 0 snap -1 2024-09-13T13:51:35.881 INFO:tasks.rados.rados.0.smithi159.stdout:1529: read oid 18 snap -1 2024-09-13T13:51:35.881 INFO:tasks.rados.rados.0.smithi159.stdout:1529: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:51:35.881 INFO:tasks.rados.rados.0.smithi159.stdout:1530: snap_remove snap 30 2024-09-13T13:51:35.893 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:35.893 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:35.893 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:35.893 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:35.893 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:35.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:35.957 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:51:35.960 INFO:tasks.rados.rados.0.smithi159.stdout:1530: done (11 left) 2024-09-13T13:51:35.960 INFO:tasks.rados.rados.0.smithi159.stdout:1531: write (excl) oid 7 current snap is 38 2024-09-13T13:51:35.960 INFO:tasks.rados.rados.0.smithi159.stdout:1531: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-13T13:51:35.966 INFO:tasks.rados.rados.0.smithi159.stdout:1531: writing smithi15938319-7 from 722767 to 1256444 tid 1 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1531: writing smithi15938319-7 from 1793793 to 2254875 tid 2 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1532: read oid 28 snap -1 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1532: expect deleted 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1533: write (excl) oid 13 current snap is 38 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1517: finishing write tid 2 to smithi15938319-33 2024-09-13T13:51:35.972 INFO:tasks.rados.rados.0.smithi159.stdout:1517: finishing write tid 3 to smithi15938319-33 2024-09-13T13:51:35.973 INFO:tasks.rados.rados.0.smithi159.stdout:1533: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-13T13:51:35.975 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:35.975+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:35.980 INFO:tasks.rados.rados.0.smithi159.stdout:1533: writing smithi15938319-13 from 739574 to 1464853 tid 1 2024-09-13T13:51:35.987 INFO:tasks.rados.rados.0.smithi159.stdout:1533: writing smithi15938319-13 from 1968628 to 2689721 tid 2 2024-09-13T13:51:35.987 INFO:tasks.rados.rados.0.smithi159.stdout:1534: read oid 24 snap -1 2024-09-13T13:51:35.988 INFO:tasks.rados.rados.0.smithi159.stdout:1534: expect (ObjNum 245 snap 28 seq_num 245) 2024-09-13T13:51:35.988 INFO:tasks.rados.rados.0.smithi159.stdout:1535: write (excl) oid 37 current snap is 38 2024-09-13T13:51:35.988 INFO:tasks.rados.rados.0.smithi159.stdout:1535: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-13T13:51:35.993 INFO:tasks.rados.rados.0.smithi159.stdout:1535: writing smithi15938319-37 from 756381 to 1156909 tid 1 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1535: writing smithi15938319-37 from 1910757 to 2375508 tid 2 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1517: finishing write tid 4 to smithi15938319-33 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1517: oid 33 updating version 0 to 201 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1517: oid 33 version 201 is already newer than 199 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1517: oid 33 version 201 is already newer than 200 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 201 (ObjNum 278 snap 38 seq_num 278) dirty exists 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1517: left oid 33 (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:36.001 INFO:tasks.rados.rados.0.smithi159.stdout:1520: expect (ObjNum 86 snap 5 seq_num 86) 2024-09-13T13:51:36.012 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.012 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:36.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.469108191, 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:36.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:51:36.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.41478230200001, 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:51:36.019 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.020 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:51:36.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.025 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.40140391200001, 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:51:36.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:36.030 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 77.588070974999994, 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:36.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782658968000007, 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.035 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:51:36.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:36.040 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782621796000001, 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:51:36.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.046 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782601068000005, 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:36.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:51:36.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782504767000006, 2024-09-13T13:51:36.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:36.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:51:36.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782482122999994, 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.059 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:36.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:51:36.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:51:36.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.782456092999993, 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.065 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:51:36.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.070 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.642955225999998, 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:51:36.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.642848279000006, 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:51:36.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:51:36.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.083 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.642819020000005, 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:51:36.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.636454616999998, 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.090 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:51:36.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:51:36.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:51:36.098 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.631494423000007, 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:51:36.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:51:36.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.689504706000001, 2024-09-13T13:51:36.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:51:36.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:36.113 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.689465568000003, 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:51:36.118 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 34.148035571000001, 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 34.147972625000001, 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:51:36.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.128 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 34.147938943, 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:51:36.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:51:36.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:51:36.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.134 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.134 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.134 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.135 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:36.017+0000 7f4ffbd7a640 -1 osd.1 117 handle_osd_map injecting map crc failure 2024-09-13T13:51:36.135 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:36.017+0000 7f002dbdb640 -1 osd.15 117 handle_osd_map injecting map crc failure 2024-09-13T13:51:36.136 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:36.134+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:36.136 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2821820 2024-09-13T13:51:36.136 INFO:tasks.rados.rados.0.smithi159.stdout:1521: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:36.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:36.216 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:36.235 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:36.234+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:36.240 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.240 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:36.240 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:36.240 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:36.259 INFO:tasks.rados.rados.0.smithi159.stdout:1518: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:51:36.332 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1717075 2024-09-13T13:51:36.338 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:36.337+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:36.351 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.352 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:36.352 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:36.352 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:36.352 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.352 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 871875 2024-09-13T13:51:36.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:36.385 INFO:tasks.rados.rados.0.smithi159.stdout:1522: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:36.413 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:36.438 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:36.437+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:36.466 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.466 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:36.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:36.466 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.727383863, 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:51:36.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:51:36.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.722143240999998, 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:51:36.473 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:51:36.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:51:36.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:51:36.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:51:36.479 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.722091926999994, 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.480 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:51:36.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:51:36.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.486 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.72201973, 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:51:36.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:51:36.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:51:36.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.721980724999995, 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:51:36.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:51:36.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:51:36.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.721916312999994, 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.501 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:51:36.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:51:36.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.696566216999997, 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:51:36.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.666005393000006, 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.513 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:51:36.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:51:36.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.644678806000002, 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.519 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:51:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:51:36.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:36.523 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.566462529000006, 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:51:36.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:51:36.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222490868999998, 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:51:36.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:51:36.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:51:36.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.535 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222432816999998, 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:36.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:51:36.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222406857999999, 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:51:36.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.548 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222292441999997, 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:51:36.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.555 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222261551000003, 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:51:36.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:36.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:51:36.562 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.222224967999999, 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:51:36.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:51:36.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.142430097999998, 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:51:36.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:51:36.575 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.767769680000001, 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:36.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.579 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 35.644578156999998, 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:51:36.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:36.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.585 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 35.636557859, 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:51:36.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.589 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.590 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.590 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.590 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.590 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.591 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:36.590+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:36.591 INFO:tasks.rados.rados.0.smithi159.stdout:1523: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:36.592 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 525807 2024-09-13T13:51:36.592 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:36.615 INFO:tasks.rados.rados.0.smithi159.stdout:1526: expect (ObjNum 221 snap 24 seq_num 221) 2024-09-13T13:51:36.630 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1966943 2024-09-13T13:51:36.679 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:36.693 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:36.691+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:36.695 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.695 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:36.695 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:36.696 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:36.794 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:36.793+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:36.806 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.807 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:36.807 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:36.807 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:36.807 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:36.815 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:36.821 INFO:tasks.rados.rados.0.smithi159.stdout:1525: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:36.873 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 346515 2024-09-13T13:51:36.894 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:36.892+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:36.920 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.690607977, 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:51:36.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:51:36.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.927 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.185869795000002, 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:51:36.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:51:36.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.933 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.185820483000001, 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.934 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:36.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:51:36.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.185804778999994, 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:51:36.940 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:51:36.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.946 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.185777103000007, 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:51:36.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:51:36.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.150559505000004, 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.953 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:51:36.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:51:36.958 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.143490522999997, 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:36.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.108952060999997, 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:51:36.965 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:51:36.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:51:36.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:51:36.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.101322218999996, 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.973 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:51:36.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:51:36.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.979 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.097945847000005, 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.980 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:51:36.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 83.995086567000001, 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:51:36.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:51:36.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.784063027000002, 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:36.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:36.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:51:36.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:36.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.772547743999993, 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:36.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:36.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.671239276000001, 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.001 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:51:37.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:51:37.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:51:37.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:37.006 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.671174927999999, 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:37.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:51:37.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.671146866000001, 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:51:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:51:37.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:51:37.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.669464976, 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:51:37.018 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:51:37.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:51:37.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:51:37.025 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.229936768000002, 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:37.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 36.101301012999997, 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:51:37.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:51:37.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:51:37.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.9560444129999999, 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:37.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:37.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:37.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:51:37.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.042 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.043 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.043 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:37.043 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:37.043 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:37.044 INFO:tasks.rados.rados.0.smithi159.stdout:1527: finishing write tid 1 to smithi15938319-36 2024-09-13T13:51:37.044 INFO:tasks.rados.rados.0.smithi159.stdout:1527: finishing write tid 2 to smithi15938319-36 2024-09-13T13:51:37.044 INFO:tasks.rados.rados.0.smithi159.stdout:1527: finishing write tid 3 to smithi15938319-36 2024-09-13T13:51:37.044 INFO:tasks.rados.rados.0.smithi159.stdout:1527: finishing write tid 4 to smithi15938319-36 2024-09-13T13:51:37.044 INFO:tasks.rados.rados.0.smithi159.stdout:1527: oid 36 updating version 0 to 202 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:1527: oid 36 updating version 202 to 203 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:1527: oid 36 updating version 203 to 204 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 204 (ObjNum 280 snap 38 seq_num 280) dirty exists 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:1527: left oid 36 (ObjNum 280 snap 38 seq_num 280) 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:1528: finishing set_chunk to oid 3 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 103424 length: 782336 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 0 (ObjNum 254 snap 30 seq_num 254) dirty exists 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:1529: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:51:37.045 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:37.045 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:37.043+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:37.046 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:37.069 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 717265 2024-09-13T13:51:37.145 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:37.144+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:37.151 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:37.151 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:37.152 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:37.152 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:37.160 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:37.185 INFO:tasks.rados.rados.0.smithi159.stdout:1531: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:37.185 INFO:tasks.rados.rados.0.smithi159.stdout:1531: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: oid 7 updating version 0 to 231 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: oid 7 updating version 231 to 232 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: oid 7 updating version 232 to 233 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 233 (ObjNum 281 snap 38 seq_num 281) dirty exists 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1531: left oid 7 (ObjNum 281 snap 38 seq_num 281) 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1533: finishing write tid 1 to smithi15938319-13 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1517: done (15 left) 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1518: done (14 left) 2024-09-13T13:51:37.186 INFO:tasks.rados.rados.0.smithi159.stdout:1520: done (13 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1521: done (12 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1522: done (11 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1523: done (10 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1525: done (9 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1526: done (8 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1527: done (7 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1528: done (6 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1529: done (5 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1531: done (4 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1532: done (3 left) 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1536: write oid 7 current snap is 38 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1533: finishing write tid 2 to smithi15938319-13 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1533: finishing write tid 3 to smithi15938319-13 2024-09-13T13:51:37.187 INFO:tasks.rados.rados.0.smithi159.stdout:1536: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-13T13:51:37.191 INFO:tasks.rados.rados.0.smithi159.stdout:1536: writing smithi15938319-7 from 773188 to 1248965 tid 1 2024-09-13T13:51:37.200 INFO:tasks.rados.rados.0.smithi159.stdout:1536: writing smithi15938319-7 from 1969239 to 2694001 tid 2 2024-09-13T13:51:37.200 INFO:tasks.rados.rados.0.smithi159.stdout:1537: read oid 17 snap -1 2024-09-13T13:51:37.200 INFO:tasks.rados.rados.0.smithi159.stdout:1537: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:37.200 INFO:tasks.rados.rados.0.smithi159.stdout:1538: read oid 43 snap -1 2024-09-13T13:51:37.200 INFO:tasks.rados.rados.0.smithi159.stdout:1538: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1539: read oid 35 snap -1 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1539: expect deleted 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1540: read oid 40 snap -1 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1540: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1533: finishing write tid 4 to smithi15938319-13 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1533: oid 13 updating version 0 to 126 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1533: oid 13 version 126 is already newer than 122 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1533: oid 13 version 126 is already newer than 125 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 126 (ObjNum 282 snap 38 seq_num 282) dirty exists 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1533: left oid 13 (ObjNum 282 snap 38 seq_num 282) 2024-09-13T13:51:37.201 INFO:tasks.rados.rados.0.smithi159.stdout:1534: expect (ObjNum 245 snap 28 seq_num 245) 2024-09-13T13:51:37.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:37.246 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:37.245+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:37.265 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:37.293 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:37.293 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:37.293 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:37.293 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:37.293 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:37.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:37.331 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 194115 2024-09-13T13:51:37.347 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:37.345+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:37.371 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:37.393 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:37.394 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1535: finishing write tid 1 to smithi15938319-37 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1535: finishing write tid 2 to smithi15938319-37 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1541: read oid 16 snap -1 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1541: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1533: done (8 left) 2024-09-13T13:51:37.397 INFO:tasks.rados.rados.0.smithi159.stdout:1534: done (7 left) 2024-09-13T13:51:37.398 INFO:tasks.rados.rados.0.smithi159.stdout:1542: write (excl) oid 45 current snap is 38 2024-09-13T13:51:37.398 INFO:tasks.rados.rados.0.smithi159.stdout:1535: finishing write tid 3 to smithi15938319-37 2024-09-13T13:51:37.398 INFO:tasks.rados.rados.0.smithi159.stdout:1542: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-13T13:51:37.404 INFO:tasks.rados.rados.0.smithi159.stdout:1542: writing smithi15938319-45 from 789995 to 1341021 tid 1 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1542: writing smithi15938319-45 from 2027721 to 2496141 tid 2 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1543: read oid 26 snap -1 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1543: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1535: finishing write tid 4 to smithi15938319-37 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1535: oid 37 updating version 0 to 186 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1535: oid 37 version 186 is already newer than 185 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1535: oid 37 updating version 186 to 187 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 187 (ObjNum 283 snap 38 seq_num 283) dirty exists 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1535: left oid 37 (ObjNum 283 snap 38 seq_num 283) 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1536: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:37.412 INFO:tasks.rados.rados.0.smithi159.stdout:1544: read oid 41 snap -1 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1544: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1535: done (9 left) 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1545: read oid 5 snap 36 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1545: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1546: read oid 14 snap -1 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1546: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1547: write oid 3 current snap is 38 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1536: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1536: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:37.413 INFO:tasks.rados.rados.0.smithi159.stdout:1536: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:1536: oid 7 updating version 0 to 235 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:1536: oid 7 updating version 235 to 236 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:1536: oid 7 version 236 is already newer than 234 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 236 (ObjNum 284 snap 38 seq_num 284) dirty exists 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:1536: left oid 7 (ObjNum 284 snap 38 seq_num 284) 2024-09-13T13:51:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:1540: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:37.438 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:37.440 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:37.440 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:37.448 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:37.447+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:37.449 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:37.458 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:37.480 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:37.525 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:37.549 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:37.548+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:37.552 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:37.552 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:37.553 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:37.553 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:37.561 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:37.581 INFO:tasks.rados.rados.0.smithi159.stdout:1538: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:37.582 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 455751 2024-09-13T13:51:37.597 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:37.598 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:37.618 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:37.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:37.639 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:37.639 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:37.643 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:37.650 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:37.649+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:37.661 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:37.661 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:37.664 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:37.664 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:37.673 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:37.682 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:37.682 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:37.703 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:37.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:37.723 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:37.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:37.749+0000 7f1db95db640 0 -- 172.21.15.27:0/1073045321 >> v2:172.21.15.122:6808/1255553857 conn(0x7f1db421f950 msgr2=0x7f1db4221da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:37.751 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:37.750+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:37.772 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:37.772 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:37.775 INFO:tasks.rados.rados.0.smithi159.stdout:1537: expect (ObjNum 232 snap 26 seq_num 232) 2024-09-13T13:51:37.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:37.818 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:37.819 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:37.827 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:37.848 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 439862 2024-09-13T13:51:37.852 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:37.851+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:37.930 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:37.930 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:37.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:37.953 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:37.952+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:37.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:51:37.989 INFO:tasks.rados.rados.0.smithi159.stdout:1547: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-13T13:51:37.994 INFO:tasks.rados.rados.0.smithi159.stdout:1547: writing smithi15938319-3 from 406802 to 1033077 tid 1 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1547: writing smithi15938319-3 from 1686203 to 2414634 tid 2 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1536: done (11 left) 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1537: done (10 left) 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1538: done (9 left) 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1539: done (8 left) 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1540: done (7 left) 2024-09-13T13:51:38.003 INFO:tasks.rados.rados.0.smithi159.stdout:1548: read oid 2 snap 28 2024-09-13T13:51:38.004 INFO:tasks.rados.rados.0.smithi159.stdout:1548: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:38.004 INFO:tasks.rados.rados.0.smithi159.stdout:1549: write (excl) oid 1 current snap is 38 2024-09-13T13:51:38.004 INFO:tasks.rados.rados.0.smithi159.stdout:1549: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-13T13:51:38.008 INFO:tasks.rados.rados.0.smithi159.stdout:1549: writing smithi15938319-1 from 423609 to 1125133 tid 1 2024-09-13T13:51:38.014 INFO:tasks.rados.rados.0.smithi159.stdout:1549: writing smithi15938319-1 from 1628332 to 2100421 tid 2 2024-09-13T13:51:38.018 INFO:tasks.rados.rados.0.smithi159.stdout:1549: writing smithi15938319-1 from 2648987 to 2823609 tid 3 2024-09-13T13:51:38.018 INFO:tasks.rados.rados.0.smithi159.stdout:1550: read oid 33 snap -1 2024-09-13T13:51:38.019 INFO:tasks.rados.rados.0.smithi159.stdout:1550: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:38.019 INFO:tasks.rados.rados.0.smithi159.stdout:1551: write (excl) oid 17 current snap is 38 2024-09-13T13:51:38.019 INFO:tasks.rados.rados.0.smithi159.stdout:1543: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:38.019 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:38.054 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:38.054+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:38.088 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.088 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.097 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:38.122 INFO:tasks.rados.rados.0.smithi159.stdout:1544: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:38.128 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1553736 2024-09-13T13:51:38.156 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:38.155+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:38.207 INFO:tasks.rados.rados.0.smithi159.stdout:1545: expect (ObjNum 194 snap 18 seq_num 194) 2024-09-13T13:51:38.226 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.226 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.235 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:38.256 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:38.256+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:38.357 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:38.357+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:38.379 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.380 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.385 INFO:tasks.rados.rados.0.smithi159.stdout:1541: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:38.388 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:38.473 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:38.472+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:38.533 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.534 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: finishing write tid 1 to smithi15938319-45 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: finishing write tid 2 to smithi15938319-45 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: finishing write tid 3 to smithi15938319-45 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: finishing write tid 4 to smithi15938319-45 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: oid 45 updating version 0 to 25 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: oid 45 updating version 25 to 26 2024-09-13T13:51:38.539 INFO:tasks.rados.rados.0.smithi159.stdout:1542: oid 45 updating version 26 to 27 2024-09-13T13:51:38.540 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 27 (ObjNum 285 snap 38 seq_num 285) dirty exists 2024-09-13T13:51:38.540 INFO:tasks.rados.rados.0.smithi159.stdout:1542: left oid 45 (ObjNum 285 snap 38 seq_num 285) 2024-09-13T13:51:38.540 INFO:tasks.rados.rados.0.smithi159.stdout:1546: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:51:38.542 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:38.560 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:38.559+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:38.583 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2504399 2024-09-13T13:51:38.592 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:38.657 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:38.660 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:38.659+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:38.686 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.686 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:38.723 INFO:tasks.rados.rados.0.smithi159.stdout:1547: finishing write tid 1 to smithi15938319-3 2024-09-13T13:51:38.723 INFO:tasks.rados.rados.0.smithi159.stdout:1547: finishing write tid 2 to smithi15938319-3 2024-09-13T13:51:38.723 INFO:tasks.rados.rados.0.smithi159.stdout:1551: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-13T13:51:38.729 INFO:tasks.rados.rados.0.smithi159.stdout:1551: writing smithi15938319-17 from 440416 to 1217189 tid 1 2024-09-13T13:51:38.735 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:51:38.736 INFO:tasks.rados.rados.0.smithi159.stdout:1551: writing smithi15938319-17 from 1686814 to 2418914 tid 2 2024-09-13T13:51:38.736 INFO:tasks.rados.rados.0.smithi159.stdout:1541: done (10 left) 2024-09-13T13:51:38.736 INFO:tasks.rados.rados.0.smithi159.stdout:1542: done (9 left) 2024-09-13T13:51:38.736 INFO:tasks.rados.rados.0.smithi159.stdout:1543: done (8 left) 2024-09-13T13:51:38.737 INFO:tasks.rados.rados.0.smithi159.stdout:1544: done (7 left) 2024-09-13T13:51:38.737 INFO:tasks.rados.rados.0.smithi159.stdout:1545: done (6 left) 2024-09-13T13:51:38.737 INFO:tasks.rados.rados.0.smithi159.stdout:1546: done (5 left) 2024-09-13T13:51:38.737 INFO:tasks.rados.rados.0.smithi159.stdout:1552: snap_remove snap 25 2024-09-13T13:51:38.761 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:38.759+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:38.810 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.810 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:38.830 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:38.829+0000 7f002dbdb640 -1 osd.15 118 handle_osd_map injecting map crc failure 2024-09-13T13:51:38.861 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:38.860+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:38.926 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:38.927 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:38.935 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:38.962 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:38.961+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:39.043 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.043 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:51:39.063 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:39.063+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:39.159 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.159 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.164 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:39.164+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:39.167 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T13:51:39.265 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:39.265+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:39.274 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.274 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T13:51:39.366 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:39.366+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:39.391 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.391 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.400 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:39.468 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:39.467+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:39.507 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.507 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:39.568 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:39.568+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:39.622 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.622 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:39.669 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:39.668+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:39.738 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.738 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.740 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:39.739+0000 7f2282f0c640 -1 osd.10 119 handle_osd_map injecting map crc failure 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1552: done (5 left) 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1553: read oid 29 snap -1 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1553: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1554: read oid 50 snap -1 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1554: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:39.741 INFO:tasks.rados.rados.0.smithi159.stdout:1555: write (excl) oid 5 current snap is 38 2024-09-13T13:51:39.742 INFO:tasks.rados.rados.0.smithi159.stdout:1555: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-13T13:51:39.742 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:39.740+0000 7fe0f9e03640 -1 osd.2 119 handle_osd_map injecting map crc failure 2024-09-13T13:51:39.747 INFO:tasks.rados.rados.0.smithi159.stdout:1555: writing smithi15938319-5 from 457223 to 1025598 tid 1 2024-09-13T13:51:39.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:39.752 INFO:tasks.rados.rados.0.smithi159.stdout:1555: writing smithi15938319-5 from 1461649 to 1937407 tid 2 2024-09-13T13:51:39.757 INFO:tasks.rados.rados.0.smithi159.stdout:1555: writing smithi15938319-5 from 2420186 to 2857223 tid 3 2024-09-13T13:51:39.757 INFO:tasks.rados.rados.0.smithi159.stdout:1556: write (excl) oid 43 current snap is 38 2024-09-13T13:51:39.757 INFO:tasks.rados.rados.0.smithi159.stdout:1547: finishing write tid 3 to smithi15938319-3 2024-09-13T13:51:39.757 INFO:tasks.rados.rados.0.smithi159.stdout:1547: finishing write tid 4 to smithi15938319-3 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1547: oid 3 updating version 0 to 238 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1547: oid 3 updating version 238 to 240 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1547: oid 3 version 240 is already newer than 239 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 240 (ObjNum 286 snap 38 seq_num 286) dirty exists 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1547: left oid 3 (ObjNum 286 snap 38 seq_num 286) 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1549: finishing write tid 1 to smithi15938319-1 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1549: finishing write tid 2 to smithi15938319-1 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:1548: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:39.758 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:39.770 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:39.770+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:39.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:39.858 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.858 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.867 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:39.871 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:39.871+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:39.902 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:39.945 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:39.945 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:39.968 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:39.968 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:39.972 INFO:tasks.rados.rados.0.smithi159.stdout:1549: finishing write tid 3 to smithi15938319-1 2024-09-13T13:51:39.972 INFO:tasks.rados.rados.0.smithi159.stdout:1549: finishing write tid 4 to smithi15938319-1 2024-09-13T13:51:39.972 INFO:tasks.rados.rados.0.smithi159.stdout:1556: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-13T13:51:39.972 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:39.971+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:39.976 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:39.976 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:39.977 INFO:tasks.rados.rados.0.smithi159.stdout:1556: writing smithi15938319-43 from 474030 to 1117654 tid 1 2024-09-13T13:51:39.983 INFO:tasks.rados.rados.0.smithi159.stdout:1556: writing smithi15938319-43 from 1520131 to 2255900 tid 2 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1547: done (8 left) 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1548: done (7 left) 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1557: write (excl) oid 4 current snap is 38 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: finishing write tid 5 to smithi15938319-1 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: oid 1 updating version 0 to 145 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: oid 1 updating version 145 to 148 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: oid 1 version 148 is already newer than 147 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: oid 1 version 148 is already newer than 146 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 148 (ObjNum 287 snap 38 seq_num 287) dirty exists 2024-09-13T13:51:39.984 INFO:tasks.rados.rados.0.smithi159.stdout:1549: left oid 1 (ObjNum 287 snap 38 seq_num 287) 2024-09-13T13:51:39.985 INFO:tasks.rados.rados.0.smithi159.stdout:1550: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:39.985 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:39.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:39.990 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:39.990 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:40.014 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:40.014 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:40.015 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:40.035 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:40.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:40.072 INFO:tasks.rados.rados.0.smithi159.stdout:1551: finishing write tid 1 to smithi15938319-17 2024-09-13T13:51:40.072 INFO:tasks.rados.rados.0.smithi159.stdout:1551: finishing write tid 2 to smithi15938319-17 2024-09-13T13:51:40.072 INFO:tasks.rados.rados.0.smithi159.stdout:1557: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-13T13:51:40.073 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:40.071+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:40.078 INFO:tasks.rados.rados.0.smithi159.stdout:1557: writing smithi15938319-4 from 490837 to 1209710 tid 1 2024-09-13T13:51:40.079 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:40.084 INFO:tasks.rados.rados.0.smithi159.stdout:1557: writing smithi15938319-4 from 1862260 to 2341687 tid 2 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1557: writing smithi15938319-4 from 2875032 to 2890837 tid 3 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1549: done (7 left) 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1550: done (6 left) 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1558: rollback oid 48 current snap is 38 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 48 to 16 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1551: finishing write tid 3 to smithi15938319-17 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1551: finishing write tid 4 to smithi15938319-17 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1551: oid 17 updating version 0 to 278 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1551: oid 17 version 278 is already newer than 276 2024-09-13T13:51:40.088 INFO:tasks.rados.rados.0.smithi159.stdout:1551: oid 17 version 278 is already newer than 277 2024-09-13T13:51:40.089 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 278 (ObjNum 288 snap 38 seq_num 288) dirty exists 2024-09-13T13:51:40.089 INFO:tasks.rados.rados.0.smithi159.stdout:1551: left oid 17 (ObjNum 288 snap 38 seq_num 288) 2024-09-13T13:51:40.089 INFO:tasks.rados.rados.0.smithi159.stdout:1554: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:40.094 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:40.094 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:40.101 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:40.101 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:40.103 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:40.106 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 309463 2024-09-13T13:51:40.122 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:40.122 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:40.143 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:40.144 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1555: finishing write tid 1 to smithi15938319-5 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1555: finishing write tid 2 to smithi15938319-5 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1551: done (6 left) 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1554: done (5 left) 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1559: rollback oid 25 current snap is 38 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 25 to 18 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1555: finishing write tid 3 to smithi15938319-5 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1555: finishing write tid 4 to smithi15938319-5 2024-09-13T13:51:40.157 INFO:tasks.rados.rados.0.smithi159.stdout:1553: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:40.165 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:40.165 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:40.173 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:40.173+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:40.186 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:40.186 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:40.207 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:40.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:40.206+0000 7f873effd640 0 -- 172.21.15.27:0/1245897244 >> v2:172.21.15.100:6800/450451503 conn(0x7f874021f350 msgr2=0x7f8740221740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:40.219 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.219 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:40.219 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:40.220 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:40.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:40.256 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:40.274 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:40.273+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:40.304 INFO:tasks.rados.rados.0.smithi159.stdout:1555: finishing write tid 5 to smithi15938319-5 2024-09-13T13:51:40.304 INFO:tasks.rados.rados.0.smithi159.stdout:1555: oid 5 updating version 0 to 185 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1555: oid 5 version 185 is already newer than 184 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1555: oid 5 version 185 is already newer than 180 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1555: oid 5 version 185 is already newer than 183 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 185 (ObjNum 289 snap 38 seq_num 289) dirty exists 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1555: left oid 5 (ObjNum 289 snap 38 seq_num 289) 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1556: finishing write tid 1 to smithi15938319-43 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1553: done (5 left) 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1555: done (4 left) 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1560: rollback oid 32 current snap is 38 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 32 to 21 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1556: finishing write tid 2 to smithi15938319-43 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1556: finishing write tid 3 to smithi15938319-43 2024-09-13T13:51:40.305 INFO:tasks.rados.rados.0.smithi159.stdout:1556: finishing write tid 4 to smithi15938319-43 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1556: oid 43 updating version 0 to 214 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1556: oid 43 version 214 is already newer than 212 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1556: oid 43 version 214 is already newer than 213 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 214 (ObjNum 290 snap 38 seq_num 290) dirty exists 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1556: left oid 43 (ObjNum 290 snap 38 seq_num 290) 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: finishing write tid 1 to smithi15938319-4 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: finishing write tid 2 to smithi15938319-4 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: finishing write tid 3 to smithi15938319-4 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: finishing write tid 4 to smithi15938319-4 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: finishing write tid 5 to smithi15938319-4 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: oid 4 updating version 0 to 248 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: oid 4 updating version 248 to 250 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: oid 4 updating version 250 to 251 2024-09-13T13:51:40.306 INFO:tasks.rados.rados.0.smithi159.stdout:1557: oid 4 version 251 is already newer than 249 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 251 (ObjNum 291 snap 38 seq_num 291) dirty exists 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1557: left oid 4 (ObjNum 291 snap 38 seq_num 291) 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1558: finishing rollback tid 0 to smithi15938319-48 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1558: finishing rollback tid 1 to smithi15938319-48 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1558: finishing rollback tid 2 to smithi15938319-48 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 288 (ObjNum 162 snap 13 seq_num 162) dirty exists 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1559: finishing rollback tid 0 to smithi15938319-25 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1559: finishing rollback tid 1 to smithi15938319-25 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1556: done (4 left) 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1557: done (3 left) 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1558: done (2 left) 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1561: snap_create 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:1559: finishing rollback tid 2 to smithi15938319-25 2024-09-13T13:51:40.307 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 158 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:51:40.308 INFO:tasks.rados.rados.0.smithi159.stdout:1560: finishing rollback tid 0 to smithi15938319-32 2024-09-13T13:51:40.314 INFO:tasks.rados.rados.0.smithi159.stdout:1560: finishing rollback tid 1 to smithi15938319-32 2024-09-13T13:51:40.314 INFO:tasks.rados.rados.0.smithi159.stdout:1560: finishing rollback tid 2 to smithi15938319-32 2024-09-13T13:51:40.314 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 130 (ObjNum 161 snap 13 seq_num 161) dirty exists 2024-09-13T13:51:40.332 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.332 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:40.332 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:40.332 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:40.332 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:40.341 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:40.384 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:40.384+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:40.430 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 87.529713737999998, 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.447 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:51:40.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:51:40.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:40.453 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 87.524914209000002, 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:51:40.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:51:40.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:51:40.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 86.675566171, 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.461 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:51:40.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:40.467 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 86.370765860000006, 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:51:40.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:51:40.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:51:40.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.171268570999999, 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:51:40.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:51:40.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:51:40.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.165408622000001, 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:51:40.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:40.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:51:40.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.160463981000007, 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:51:40.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:51:40.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:51:40.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.160253402999999, 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.498 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:51:40.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.505 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.160205304000002, 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:51:40.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:51:40.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.077124893000004, 2024-09-13T13:51:40.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:51:40.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.077016181999994, 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.519 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:51:40.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.076900902000006, 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:51:40.524 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:51:40.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:51:40.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.2029298, 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.534 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.202843483000002, 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:51:40.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.202783113000002, 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:40.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 38.568930967999997, 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2634 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 0~2456909] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.623832952000001, 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087606320000000008, 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2634 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821172+0000", 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821175+0000", 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5719999999999999e-06 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821180+0000", 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821185+0000", 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6590000000000004e-06 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821322+0000", 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013670500000000001 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821328+0000", 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8170000000000004e-06 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829522+0000", 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081938299999999992 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829600+0000", 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7750000000000006e-05 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829932+0000", 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033246000000000003 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.557 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3008 4.7 4:e26a40dd:::smithi15938319-25:head [sparse-read 0~3142009] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.241736521, 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0093824860000000006, 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3008 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203272+0000", 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6770000000000001e-06 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203280+0000", 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203287+0000", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210677+0000", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073896709999999996 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210743+0000", 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6392999999999999e-05 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.212650+0000", 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019070300000000001 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3018 4.7 4:e63fe140:::smithi15938319-1:14 [read 792552~2165480,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.229795277000001, 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0088409060000000008, 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3018 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215210+0000", 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0299999999999998e-07 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215214+0000", 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4889999999999997e-06 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215228+0000", 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3448999999999999e-05 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215238+0000", 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9059999999999997e-06 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.223122+0000", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0078847910000000004 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.223193+0000", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1019999999999994e-05 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.224050+0000", 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085654800000000005 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3134 4.7 4:e1bd516b:::smithi15938319-19:head [write 769820~720742,stat] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.272090364, 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096203899999999995, 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3134 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172917+0000", 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6759999999999999e-06 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173729+0000", 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081220499999999998 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173743+0000", 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4073000000000001e-05 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173749+0000", 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173849+0000", 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9809999999999995e-05 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173917+0000", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7556000000000002e-05 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.174067+0000", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000150335 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.180026+0000", 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059590329999999999 2024-09-13T13:51:40.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182455+0000", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024288000000000001 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182498+0000", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2861000000000002e-05 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182534+0000", 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6522e-05 2024-09-13T13:51:40.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.573 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.573 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.573 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.573 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.573 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:40.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:40.575 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:40.575 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:40.576 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:40.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:40.577 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:40.573+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:40.675 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:40.675+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:40.718 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.719 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:40.719 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:40.719 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:40.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:40.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:51:40.776 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:40.775+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:40.829 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:40.830 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:40.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:40.878 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:40.877+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:40.942 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:40.942 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:40.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 87.171441913999999, 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:51:40.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:51:40.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:51:40.949 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 87.162411481999996, 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:51:40.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.510405606000006, 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:51:40.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.507371620000001, 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:40.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:51:40.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.631539501999995, 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:51:40.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:51:40.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.625632670000002, 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:51:40.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:51:40.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.618197234999997, 2024-09-13T13:51:40.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.698903493000003, 2024-09-13T13:51:40.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.978 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:51:40.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:51:40.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.984 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.698834828000003, 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:40.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:51:40.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:40.991 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.698784713999999, 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:51:40.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:51:40.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:51:40.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:40.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.677827352999998, 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.998 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:40.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:51:41.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.672231150000002, 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:51:41.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:51:41.009 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.670687239999999, 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:51:41.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.670622192000003, 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:51:41.016 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:51:41.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.021 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.063680593000001, 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.026 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.208004889999998, 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:41.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.031 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.136109202, 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.032 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:51:41.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:51:41.037 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.136071680999997, 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:51:41.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 49.023152926000002, 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.043 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:51:41.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:51:41.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.809086518000001, 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:41.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:51:41.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:51:41.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.056 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.057 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:40.987+0000 7faeaa210640 -1 osd.8 120 handle_osd_map injecting map crc failure 2024-09-13T13:51:41.058 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:41.056+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:41.058 INFO:tasks.rados.rados.0.smithi159.stdout:1559: done (2 left) 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1560: done (1 left) 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1561: done (0 left) 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1562: set_chunk oid 22 target oid 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1562: set_chunk oid 22 offset: 627200 length: 1132544 target oid e7d27e36dc298e6bc1deea5dc469dbe8af60eda37fe7f536910f9dfc15836317 offset: 0 snap -1 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1563: snap_create 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:1562: finishing set_chunk to oid 22 2024-09-13T13:51:41.059 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 31 (ObjNum 277 snap 38 seq_num 277) dirty exists 2024-09-13T13:51:41.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:41.158 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:41.158+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:41.163 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.163 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:41.163 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:41.163 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:41.259 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:41.258+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:41.274 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.275 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:41.275 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:41.275 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:41.275 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:41.361 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:41.360+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:41.385 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.150634172, 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:51:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 109.142600026, 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:51:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:51:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 59.746901866999998, 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:51:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.199998165, 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:51:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.195157341000002, 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:51:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:51:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.194444365999999, 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:51:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:51:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.416 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:41.462 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:41.461+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:41.520 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.521 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:41.521 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:41.521 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.529 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:41.563 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:41.561+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:41.632 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.632 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:41.632 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:41.632 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:41.632 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:41.684 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:41.684+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:41.744 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.975482251000003, 2024-09-13T13:51:41.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:51:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:51:41.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:51:41.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.974505053000001, 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:51:41.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.972750387999994, 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.759 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:51:41.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:51:41.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.972358174999997, 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:41.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:51:41.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:51:41.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:51:41.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:51:41.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:51:41.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.972320847999995, 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.776 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:51:41.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:51:41.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.967522494999997, 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:41.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:51:41.788 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.95188727, 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:51:41.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:51:41.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.796 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.951780920000004, 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.797 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:51:41.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:51:41.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.803 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.943847149999996, 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:41.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:51:41.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:51:41.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:51:41.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:241 3.1 3:a0105280:::smithi15938319-42 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:41.810 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.925584405999999, 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016031050000000002, 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 241 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816701+0000", 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816704+0000", 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2170000000000001e-06 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816709+0000", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8830000000000003e-06 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816722+0000", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2539e-05 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816730+0000", 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0779999999999996e-06 2024-09-13T13:51:41.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816788+0000", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7537999999999997e-05 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816960+0000", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172836 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.825142+0000", 2024-09-13T13:51:41.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081820799999999996 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.829348+0000", 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042059640000000004 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832655+0000", 2024-09-13T13:51:41.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003306752 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832697+0000", 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2114999999999999e-05 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.832732+0000", 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5048000000000001e-05 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.925458591999998, 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:51:41.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.925331802000002, 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.824 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:51:41.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.844133378999999, 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.830 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:51:41.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.844031928000007, 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.837 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:51:41.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 77.282679161999994, 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.843 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:51:41.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:51:41.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 77.276827619000002, 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:51:41.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 73.000017983000006, 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:51:41.854 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:41.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:51:41.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:51:41.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.014950167999999, 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.859 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:51:41.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:51:41.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:51:41.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.008928893000004, 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2394 3.1 3:93ff5947:::3634ac6462bf9cb96fe06c81293bb3c6c0e95892f99c8f55c0a14944e78c533e:head [write 0~1120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.963541036000002, 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012999503000000001, 2024-09-13T13:51:41.872 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2394 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778745+0000", 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.778747+0000", 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.886e-06 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:41.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780479+0000", 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017320300000000001 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780496+0000", 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7070000000000001e-05 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780522+0000", 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6588999999999999e-05 2024-09-13T13:51:41.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780613+0000", 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0382000000000003e-05 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.780619+0000", 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.353e-06 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:41.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786863+0000", 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062442749999999997 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.786980+0000", 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116236 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.787126+0000", 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014658299999999999 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791421+0000", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042944840000000003 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791672+0000", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025098399999999999 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791713+0000", 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1306000000000001e-05 2024-09-13T13:51:41.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.791744+0000", 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1325000000000002e-05 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:41.879 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:41.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:41.881 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:41.880+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:41.977 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:51:41.981 INFO:tasks.rados.rados.0.smithi159.stdout:1562: done (1 left) 2024-09-13T13:51:41.981 INFO:tasks.rados.rados.0.smithi159.stdout:1563: done (0 left) 2024-09-13T13:51:41.981 INFO:tasks.rados.rados.0.smithi159.stdout:1564: tier_promote oid 30 2024-09-13T13:51:41.981 INFO:tasks.rados.rados.0.smithi159.stdout:1565: set_chunk oid 1 target oid 2024-09-13T13:51:41.981 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:41.980+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:41.997 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:41.997 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:41.998 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:41.998 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.002 INFO:tasks.rados.rados.0.smithi159.stdout:1565: set_chunk oid 1 offset: 183296 length: 1551872 target oid 75ea54bd936a418d2ce844c97eb8f6986b626600bcf4a038e70063618c3be473 offset: 0 snap 38 2024-09-13T13:51:42.002 INFO:tasks.rados.rados.0.smithi159.stdout:1566: set_chunk oid 26 target oid 2024-09-13T13:51:42.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:42.016 INFO:tasks.rados.rados.0.smithi159.stdout:1566: set_chunk oid 26 offset: 712192 length: 1361920 target oid ae8b86d07d375a083683440b6bda0e0f67b5195d585764d7ac407424ac114e59 offset: 0 snap 36 2024-09-13T13:51:42.016 INFO:tasks.rados.rados.0.smithi159.stdout:1567: snap_create 2024-09-13T13:51:42.016 INFO:tasks.rados.rados.0.smithi159.stdout:1564: got (0) Success 2024-09-13T13:51:42.016 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 179 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-09-13T13:51:42.016 INFO:tasks.rados.rados.0.smithi159.stdout:1565: finishing set_chunk to oid 1 2024-09-13T13:51:42.082 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:42.081+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:42.110 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:42.110 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:42.111 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:42.111 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:42.111 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:42.182 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:42.181+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 89.363050360000003, 2024-09-13T13:51:42.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:42.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:51:42.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:51:42.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 89.347422745000003, 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.270 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:51:42.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:51:42.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:42.276 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 89.338711070000002, 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:51:42.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:51:42.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:315 4.1 4:8b8235b2:::smithi15938319-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 88.491165107, 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088218680000000001, 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 315 2024-09-13T13:51:42.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770051+0000", 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770052+0000", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770056+0000", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770070+0000", 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3509999999999999e-05 2024-09-13T13:51:42.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770078+0000", 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2749999999999995e-06 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770128+0000", 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0179999999999997e-05 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.770205+0000", 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7033000000000006e-05 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.777683+0000", 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074770390000000004 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778598+0000", 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091583099999999998 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778727+0000", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128708 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778792+0000", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5403999999999999e-05 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.778873+0000", 2024-09-13T13:51:42.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0154999999999994e-05 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 73.514424919999996, 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.289 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:51:42.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 72.299754641000007, 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:51:42.294 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:51:42.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.297953015000004, 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.300 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:51:42.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:42.304 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.019029447999998, 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:51:42.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:51:42.309 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.019006089000001, 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:51:42.310 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.311 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:51:42.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:51:42.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.316 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 50.520561252, 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.317 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:51:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:51:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 41.434939888999999, 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:51:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 31.500069645, 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:51:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2853 4.1 4:8e40678c:::smithi15938319-17:head [sparse-read 0~2202141,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.404185228999999, 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098372819999999993, 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2853 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857033+0000", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3149999999999999e-06 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857041+0000", 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5369999999999997e-06 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857060+0000", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8972000000000001e-05 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857070+0000", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3489999999999998e-06 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857145+0000", 2024-09-13T13:51:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4870000000000007e-05 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857153+0000", 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5310000000000001e-06 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.862925+0000", 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057717510000000003 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.863022+0000", 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6618000000000002e-05 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866868+0000", 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038463389999999998 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.404125329999999, 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:51:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.404087469, 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:51:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.423392598, 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:51:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:51:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:51:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.422999873999998, 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:51:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:51:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.422984274000001, 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:51:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:51:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:51:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3367 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.2882149089999997, 2024-09-13T13:51:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096522090000000001, 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3367 2024-09-13T13:51:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973002+0000", 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2799999999999995e-07 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973007+0000", 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9799999999999998e-06 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973015+0000", 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973023+0000", 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973140+0000", 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011665100000000001 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973146+0000", 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4960000000000001e-06 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980078+0000", 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069321570000000004 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980132+0000", 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3834e-05 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.982653+0000", 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025209910000000002 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3399 4.1 4:80c58c2d:::smithi15938319-35:head [delete] snapc 27=[27,26,25,24,23,22,20,1e,1b,1a,17,14,12] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-13T13:51:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.2526343779999998, 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088886150000000008, 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3399 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008582+0000", 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4300000000000003e-07 2024-09-13T13:51:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008584+0000", 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3410000000000001e-06 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008591+0000", 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4320000000000004e-06 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008598+0000", 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015266+0000", 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066677849999999999 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015323+0000", 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7108000000000003e-05 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015457+0000", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013399700000000001 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017064+0000", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016070769999999999 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017414+0000", 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00034987400000000002 2024-09-13T13:51:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017445+0000", 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0917999999999998e-05 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017470+0000", 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5576e-05 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.383 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:42.385 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:42.384+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:42.485 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:42.484+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:42.487 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:42.488 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:42.488 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:51:42.488 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.496 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:42.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:42.599 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:42.599 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:51:42.599 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:51:42.600 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:51:42.600 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:42.622 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:51:42.621+0000 7f1f44d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33214) UID: 0 2024-09-13T13:51:42.687 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:42.686+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:42.713 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 114.169418898, 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.714 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:51:42.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:51:42.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.11509300900001, 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:51:42.722 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:51:42.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:51:42.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:42.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:51:42.728 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.10171461900001, 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:51:42.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.732 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 84.288381681999994, 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.733 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:51:42.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.737 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482969675000007, 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:51:42.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:51:42.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.743 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482932503000001, 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:42.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:51:42.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.749 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482911775000005, 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.750 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:51:42.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:51:42.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482815474000006, 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.756 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:51:42.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482792829999994, 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.762 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:51:42.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:51:42.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:42.768 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.482766799999993, 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:42.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:51:42.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.774 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.343265932999998, 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:51:42.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:51:42.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:42.780 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.343158986000006, 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:42.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:51:42.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.786 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.343129727000004, 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:51:42.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.336765323999998, 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.793 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:51:42.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:51:42.800 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.331805130000006, 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:51:42.801 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:51:42.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:51:42.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.389815412999994, 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:51:42.810 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:51:42.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:51:42.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:42.816 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 67.389776275000003, 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:51:42.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.821 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 40.848346278000001, 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.822 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 40.848283332000001, 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.827 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.828 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:51:42.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:51:42.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 40.84824965, 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:42.832 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:51:42.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:42.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:42.837 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:42.838 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:42.839 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:42.839 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:42.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:42.840 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:42.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:42.841 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4: {} 2024-09-13T13:51:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T13:51:42.842 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:42.842 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:42.842 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:42.837+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:42.856 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:42.856 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:42.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:42.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:42.898 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:42.898 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:42.914 INFO:tasks.rados.rados.0.smithi159.stdout:1566: finishing set_chunk to oid 26 2024-09-13T13:51:42.924 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:42.924 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:42.939 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:42.938+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:42.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:42.944 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:42.966 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:42.966 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:42.983 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:42.984 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:42.987 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:42.987 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:42.988 INFO:tasks.rados.rados.0.smithi159.stdout:1564: done (3 left) 2024-09-13T13:51:42.989 INFO:tasks.rados.rados.0.smithi159.stdout:1565: done (2 left) 2024-09-13T13:51:42.989 INFO:tasks.rados.rados.0.smithi159.stdout:1566: done (1 left) 2024-09-13T13:51:42.989 INFO:tasks.rados.rados.0.smithi159.stdout:1567: done (0 left) 2024-09-13T13:51:42.989 INFO:tasks.rados.rados.0.smithi159.stdout:1568: write (excl) oid 17 current snap is 41 2024-09-13T13:51:42.989 INFO:tasks.rados.rados.0.smithi159.stdout:1568: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-13T13:51:42.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:42.996 INFO:tasks.rados.rados.0.smithi159.stdout:1568: writing smithi15938319-17 from 507644 to 1301766 tid 1 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1568: writing smithi15938319-17 from 1920742 to 2543827 tid 2 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1569: read oid 25 snap -1 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1569: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1570: read oid 49 snap -1 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1570: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1571: read oid 47 snap -1 2024-09-13T13:51:43.003 INFO:tasks.rados.rados.0.smithi159.stdout:1571: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1572: tier_evict oid 33 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1572: tier_evict oid 33 snap -1 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1573: read oid 44 snap -1 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1573: expect (ObjNum 210 snap 22 seq_num 210) 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1574: read oid 37 snap 36 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1574: expect deleted 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1575: write oid 19 current snap is 41 2024-09-13T13:51:43.004 INFO:tasks.rados.rados.0.smithi159.stdout:1575: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-13T13:51:43.008 INFO:tasks.rados.rados.0.smithi159.stdout:1575: writing smithi15938319-19 from 524451 to 993822 tid 1 2024-09-13T13:51:43.011 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:43.014 INFO:tasks.rados.rados.0.smithi159.stdout:1575: writing smithi15938319-19 from 1579224 to 2062320 tid 2 2024-09-13T13:51:43.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:43.016+0000 7fc93cdaa640 0 -- 172.21.15.27:0/2125276054 >> v2:172.21.15.122:6808/1255553857 conn(0x7fc93821fd90 msgr2=0x7fc938222180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:43.019 INFO:tasks.rados.rados.0.smithi159.stdout:1575: writing smithi15938319-19 from 2529878 to 2924451 tid 3 2024-09-13T13:51:43.019 INFO:tasks.rados.rados.0.smithi159.stdout:1576: write (excl) oid 7 current snap is 41 2024-09-13T13:51:43.019 INFO:tasks.rados.rados.0.smithi159.stdout:1576: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-13T13:51:43.020 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-09-13T13:51:43.020 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T13:51:43.021 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T13:51:43.021 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:51:43.025 INFO:tasks.rados.rados.0.smithi159.stdout:1576: writing smithi15938319-7 from 541258 to 1085878 tid 1 2024-09-13T13:51:43.031 INFO:tasks.rados.rados.0.smithi159.stdout:1576: writing smithi15938319-7 from 1637706 to 2264460 tid 2 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1576: writing smithi15938319-7 from 2840948 to 2941258 tid 3 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1577: read oid 3 snap -1 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1577: expect (ObjNum 286 snap 38 seq_num 286) 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1578: tier_promote oid 35 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1568: finishing write tid 1 to smithi15938319-17 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1579: read oid 23 snap -1 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1579: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1580: read oid 41 snap -1 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1580: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1581: tier_evict oid 21 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1581: tier_evict oid 21 snap -1 2024-09-13T13:51:43.036 INFO:tasks.rados.rados.0.smithi159.stdout:1572: got (22) Invalid argument 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: finishing write tid 2 to smithi15938319-17 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: finishing write tid 3 to smithi15938319-17 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: finishing write tid 4 to smithi15938319-17 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: oid 17 updating version 0 to 291 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: oid 17 updating version 291 to 292 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: oid 17 version 292 is already newer than 290 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 292 (ObjNum 292 snap 41 seq_num 292) dirty exists 2024-09-13T13:51:43.037 INFO:tasks.rados.rados.0.smithi159.stdout:1568: left oid 17 (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:51:43.038 INFO:tasks.rados.rados.0.smithi159.stdout:1570: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:51:43.040 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:43.040 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:43.041 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:43.039+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:43.053 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T13:51:43.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 4 2024-09-13T13:51:43.064 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:43.064 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:43.070 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:43.102 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.102 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:43.141 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:43.140+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:43.158 INFO:tasks.rados.rados.0.smithi159.stdout:1571: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:51:43.200 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:43.215 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.215 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.224 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:43.268 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:43.267+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:43.277 INFO:tasks.rados.rados.0.smithi159.stdout:1568: done (13 left) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1570: done (12 left) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1571: done (11 left) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1572: done (10 left) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1574: done (9 left) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1582: read oid 9 snap 36 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1582: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1583: tier_promote oid 27 2024-09-13T13:51:43.278 INFO:tasks.rados.rados.0.smithi159.stdout:1573: expect (ObjNum 210 snap 22 seq_num 210) 2024-09-13T13:51:43.327 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:43.329 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.329 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.338 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:43.377 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-13T13:51:43.376+0000 7fe36cfd9640 -1 mon.a@0(leader).osd e123 definitely_dead 0 2024-09-13T13:51:43.379 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:43.378+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:43.442 INFO:tasks.rados.rados.0.smithi159.stdout:1569: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:51:43.443 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:43.442+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:43.458 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:43.457+0000 7f586a715640 -1 osd.9 122 handle_osd_map injecting map crc failure 2024-09-13T13:51:43.482 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.482 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.491 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:43.543 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:43.542+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.605 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:43.627 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:43.644 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:43.643+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:43.710 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.710 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.719 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:43.724 INFO:tasks.rados.rados.0.smithi159.stdout:1575: finishing write tid 1 to smithi15938319-19 2024-09-13T13:51:43.724 INFO:tasks.rados.rados.0.smithi159.stdout:1575: finishing write tid 2 to smithi15938319-19 2024-09-13T13:51:43.724 INFO:tasks.rados.rados.0.smithi159.stdout:1575: finishing write tid 3 to smithi15938319-19 2024-09-13T13:51:43.724 INFO:tasks.rados.rados.0.smithi159.stdout:1575: finishing write tid 4 to smithi15938319-19 2024-09-13T13:51:43.724 INFO:tasks.rados.rados.0.smithi159.stdout:1575: finishing write tid 5 to smithi15938319-19 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1575: oid 19 updating version 0 to 162 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1575: oid 19 version 162 is already newer than 161 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1575: oid 19 updating version 162 to 163 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1575: oid 19 updating version 163 to 164 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 164 (ObjNum 293 snap 41 seq_num 293) dirty exists 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1575: left oid 19 (ObjNum 293 snap 41 seq_num 293) 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1581: got (22) Invalid argument 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1578: got (0) Success 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 35 v 270 (ObjNum 97 snap 0 seq_num 1) clean dne 2024-09-13T13:51:43.725 INFO:tasks.rados.rados.0.smithi159.stdout:1580: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:43.736 INFO:teuthology.orchestra.run.smithi027.stderr:marked down osd.4. 2024-09-13T13:51:43.737 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1749554 2024-09-13T13:51:43.745 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:43.744+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:43.748 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:43.751 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 6] 2024-09-13T13:51:43.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-09-13T13:51:43.810 INFO:tasks.rados.rados.0.smithi159.stdout:1579: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:43.813 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 640320 2024-09-13T13:51:43.829 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.829 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.837 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:43.846 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:43.845+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:43.936 INFO:tasks.rados.rados.0.smithi159.stdout:1576: finishing write tid 1 to smithi15938319-7 2024-09-13T13:51:43.936 INFO:tasks.rados.rados.0.smithi159.stdout:1576: finishing write tid 2 to smithi15938319-7 2024-09-13T13:51:43.936 INFO:tasks.rados.rados.0.smithi159.stdout:1569: done (10 left) 2024-09-13T13:51:43.936 INFO:tasks.rados.rados.0.smithi159.stdout:1573: done (9 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1575: done (8 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1578: done (7 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1579: done (6 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1580: done (5 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1581: done (4 left) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1584: read oid 26 snap -1 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1584: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1585: read oid 23 snap -1 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1585: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1586: read oid 42 snap -1 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1586: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1587: read oid 16 snap -1 2024-09-13T13:51:43.937 INFO:tasks.rados.rados.0.smithi159.stdout:1587: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1588: write oid 45 current snap is 41 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: finishing write tid 3 to smithi15938319-7 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: finishing write tid 4 to smithi15938319-7 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: finishing write tid 5 to smithi15938319-7 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: oid 7 updating version 0 to 254 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: oid 7 updating version 254 to 256 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: oid 7 version 256 is already newer than 255 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: oid 7 updating version 256 to 257 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 257 (ObjNum 294 snap 41 seq_num 294) dirty exists 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1576: left oid 7 (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:51:43.938 INFO:tasks.rados.rados.0.smithi159.stdout:1577: expect (ObjNum 286 snap 38 seq_num 286) 2024-09-13T13:51:43.946 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:43.946+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:43.976 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:43.975+0000 7fc16a180640 -1 osd.11 123 handle_osd_map injecting map crc failure 2024-09-13T13:51:43.986 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:43.986 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:43.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:51:43.996 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:43.997 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:44.047 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:44.046+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:44.065 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1034040 2024-09-13T13:51:44.101 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.101 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.109 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-13T13:51:44.108+0000 7fe36cfd9640 -1 mon.a@0(leader).osd e124 definitely_dead 0 2024-09-13T13:51:44.110 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:44.111 INFO:tasks.rados.rados.0.smithi159.stdout:1588: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-13T13:51:44.117 INFO:tasks.rados.rados.0.smithi159.stdout:1588: writing smithi15938319-45 from 558065 to 1177934 tid 1 2024-09-13T13:51:44.123 INFO:tasks.rados.rados.0.smithi159.stdout:1588: writing smithi15938319-45 from 1579835 to 2066600 tid 2 2024-09-13T13:51:44.128 INFO:tasks.rados.rados.0.smithi159.stdout:1588: writing smithi15938319-45 from 2468371 to 2958065 tid 3 2024-09-13T13:51:44.128 INFO:tasks.rados.rados.0.smithi159.stdout:1576: done (8 left) 2024-09-13T13:51:44.128 INFO:tasks.rados.rados.0.smithi159.stdout:1577: done (7 left) 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:1589: read oid 15 snap -1 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:1589: expect deleted 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:1590: write (excl) oid 48 current snap is 41 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:1583: got (0) Success 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 164 (ObjNum 223 snap 24 seq_num 223) dirty exists 2024-09-13T13:51:44.129 INFO:tasks.rados.rados.0.smithi159.stdout:1584: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:44.148 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:44.148+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:44.201 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1623466 2024-09-13T13:51:44.214 INFO:tasks.rados.rados.0.smithi159.stdout:1585: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:44.215 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.216 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.224 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:44.249 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:44.248+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:44.260 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:44.308 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 447822 2024-09-13T13:51:44.349 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:44.349+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:44.350 INFO:tasks.rados.rados.0.smithi159.stdout:1586: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:44.350 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 47962 2024-09-13T13:51:44.363 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.364 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.373 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:44.450 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:44.449+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:44.470 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1471959 2024-09-13T13:51:44.477 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.477 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.486 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:44.504 INFO:tasks.rados.rados.0.smithi159.stdout:1587: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:44.504 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 67719 2024-09-13T13:51:44.551 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:44.550+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:44.596 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 923418 2024-09-13T13:51:44.609 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.609 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.617 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: finishing write tid 1 to smithi15938319-45 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: finishing write tid 2 to smithi15938319-45 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: finishing write tid 3 to smithi15938319-45 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: finishing write tid 4 to smithi15938319-45 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: finishing write tid 5 to smithi15938319-45 2024-09-13T13:51:44.633 INFO:tasks.rados.rados.0.smithi159.stdout:1588: oid 45 updating version 0 to 32 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:1588: oid 45 updating version 32 to 34 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:1588: oid 45 version 34 is already newer than 33 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:1588: oid 45 version 34 is already newer than 31 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 34 (ObjNum 295 snap 41 seq_num 295) dirty exists 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:1588: left oid 45 (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:51:44.634 INFO:tasks.rados.rados.0.smithi159.stdout:1590: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-13T13:51:44.640 INFO:tasks.rados.rados.0.smithi159.stdout:1590: writing smithi15938319-48 from 574872 to 1269990 tid 1 2024-09-13T13:51:44.648 INFO:tasks.rados.rados.0.smithi159.stdout:1590: writing smithi15938319-48 from 2038317 to 2668740 tid 2 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1583: done (8 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1584: done (7 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1585: done (6 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1586: done (5 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1587: done (4 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1588: done (3 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1589: done (2 left) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1591: read oid 23 snap -1 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1591: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1592: tier_promote oid 11 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1590: finishing write tid 1 to smithi15938319-48 2024-09-13T13:51:44.649 INFO:tasks.rados.rados.0.smithi159.stdout:1593: read oid 15 snap -1 2024-09-13T13:51:44.650 INFO:tasks.rados.rados.0.smithi159.stdout:1593: expect deleted 2024-09-13T13:51:44.650 INFO:tasks.rados.rados.0.smithi159.stdout:1594: snap_create 2024-09-13T13:51:44.652 INFO:tasks.rados.rados.0.smithi159.stdout:1591: expect (ObjNum 265 snap 32 seq_num 265) 2024-09-13T13:51:44.652 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:44.651+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:44.737 INFO:teuthology.orchestra.run.smithi027.stderr:marked out osd.4. 2024-09-13T13:51:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:51:44.739 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.739 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.740 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1618122 2024-09-13T13:51:44.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:44.754 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-13T13:51:44.754 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:51:44.754 DEBUG:teuthology.orchestra.run.smithi027:> type ceph-objectstore-tool 2024-09-13T13:51:44.769 INFO:tasks.rados.rados.0.smithi159.stdout:1591: done (5 left) 2024-09-13T13:51:44.769 INFO:tasks.rados.rados.0.smithi159.stdout:1593: done (4 left) 2024-09-13T13:51:44.769 INFO:tasks.rados.rados.0.smithi159.stdout:1594: done (3 left) 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1595: read oid 12 snap -1 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1595: expect (ObjNum 270 snap 35 seq_num 270) 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1596: snap_create 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: finishing write tid 2 to smithi15938319-48 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: finishing write tid 3 to smithi15938319-48 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: finishing write tid 4 to smithi15938319-48 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: oid 48 updating version 0 to 296 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: oid 48 version 296 is already newer than 295 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: oid 48 updating version 296 to 297 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 297 (ObjNum 296 snap 41 seq_num 296) dirty exists 2024-09-13T13:51:44.770 INFO:tasks.rados.rados.0.smithi159.stdout:1590: left oid 48 (ObjNum 296 snap 41 seq_num 296) 2024-09-13T13:51:44.771 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:44.768+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:44.782 INFO:tasks.rados.rados.0.smithi159.stdout:1595: expect (ObjNum 270 snap 35 seq_num 270) 2024-09-13T13:51:44.782 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:44.809 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T13:51:44.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-13T13:51:44.819 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:44.854 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:44.853+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:44.859 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:44.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:44.955 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:44.953+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:45.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:45.055 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:45.054+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:45.076 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:45.156 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:45.154+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:45.174 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:45.174 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:45.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi027.stdout:4.e 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi027.stdout:4.0 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi027.stdout:3.1 2024-09-13T13:51:45.257 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:45.256+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:45.294 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:45.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:45.304 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:45.358 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:45.357+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:45.370 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:45.371 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:45.392 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:45.392 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:45.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:45.415 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:45.435 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:45.435 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:45.435 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:45.436 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:45.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:45.460 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:45.457+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:45.481 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:45.481 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:45.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 4.e --file /var/log/ceph/exp.4.e.4 2024-09-13T13:51:45.502 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:45.502 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:45.524 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:45.546 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:45.546 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:45.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:45.550+0000 7fa7ce7fc640 0 -- 172.21.15.27:0/2963780515 >> v2:172.21.15.159:6805/4073091144 conn(0x7fa7d0220350 msgr2=0x7fa7d0222760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:45.560 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:45.559+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:45.572 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:45.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.405185141000004, 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:51:45.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:51:45.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.405159147999996, 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:51:45.579 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:45.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.299850325999998, 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:51:45.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.003815149000005, 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:51:45.591 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.003800648999999, 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:51:45.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:51:45.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:51:45.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.001547208999995, 2024-09-13T13:51:45.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:51:45.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 92.879123921000001, 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:51:45.608 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:45.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:51:45.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:51:45.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 92.879093101999999, 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.615 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:51:45.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.620 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 92.876934993999996, 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:51:45.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.624 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 92.667400795999995, 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:45.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:51:45.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 91.801423263999993, 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.629 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:51:45.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:51:45.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:51:45.635 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 91.787747550000006, 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:51:45.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.771845966000001, 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:51:45.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.763554470999999, 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.649 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:51:45.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:45.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:51:45.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:51:45.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.759527550000001, 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:51:45.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:51:45.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:45.664 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.759202594000001, 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:51:45.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:51:45.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:51:45.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:45.672 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.759171068999997, 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:51:45.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:51:45.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:51:45.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.679 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 40.731481068000001, 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:45.680 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:51:45.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3255 4.a 4:5004af91:::smithi15938319-34:head [append 0~0] snapc 23=[23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.822898211000002, 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.285444431, 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3255 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747900+0000", 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1220000000000002e-06 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747906+0000", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5979999999999996e-06 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747911+0000", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4e-06 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750840+0000", 2024-09-13T13:51:45.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029289250000000002 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750936+0000", 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5449e-05 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751081+0000", 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014517800000000001 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.688 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751499+0000", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00041844799999999998 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033276+0000", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28177714300000001 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033312+0000", 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5370999999999997e-05 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033342+0000", 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0457999999999999e-05 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:3255 4.a e112/85 ondisk, result = 0)", 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.818259117, 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.280808787, 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:45.690 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752540+0000", 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9849999999999998e-06 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:45.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033199+0000", 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28065851600000002 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033203+0000", 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5029999999999999e-06 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033267+0000", 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3596999999999994e-05 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033273+0000", 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.789e-06 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033346+0000", 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2960999999999995e-05 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:45.694 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:45.696 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:45.696 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:45.694+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:45.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:45.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:45.743 INFO:tasks.rados.rados.0.smithi159.stdout:1590: done (4 left) 2024-09-13T13:51:45.743 INFO:tasks.rados.rados.0.smithi159.stdout:1595: done (3 left) 2024-09-13T13:51:45.743 INFO:tasks.rados.rados.0.smithi159.stdout:1596: done (2 left) 2024-09-13T13:51:45.743 INFO:tasks.rados.rados.0.smithi159.stdout:1597: snap_remove snap 21 2024-09-13T13:51:45.796 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:45.795+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:45.850 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:45.850 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:45.850 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:45.850 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:45.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:45.886 INFO:teuthology.orchestra.run.smithi027.stderr:Exporting 4.e info 4.e( v 120'239 (0'0,120'239] local-lis/les=34/35 n=12 ec=34/34 lis/c=34/34 les/c/f=35/35/0 sis=34) 2024-09-13T13:51:45.888 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:709ad093:::smithi15938319-50:12# 2024-09-13T13:51:45.891 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:709ad093:::smithi15938319-50:head# 2024-09-13T13:51:45.895 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:70cd1372:::smithi15938319-9:1f# 2024-09-13T13:51:45.897 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:45.896+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:45.899 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:70cd1372:::smithi15938319-9:head# 2024-09-13T13:51:45.904 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:717ade62:::smithi15938319-24:13# 2024-09-13T13:51:45.908 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:717ade62:::smithi15938319-24:1b# 2024-09-13T13:51:45.913 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:717ade62:::smithi15938319-24:head# 2024-09-13T13:51:45.916 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:721f6cf4:::smithi15938319-18:13# 2024-09-13T13:51:45.920 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:721f6cf4:::smithi15938319-18:14# 2024-09-13T13:51:45.925 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:721f6cf4:::smithi15938319-18:1d# 2024-09-13T13:51:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:721f6cf4:::smithi15938319-18:21# 2024-09-13T13:51:45.932 INFO:teuthology.orchestra.run.smithi027.stderr:Read #4:721f6cf4:::smithi15938319-18:head# 2024-09-13T13:51:45.936 INFO:teuthology.orchestra.run.smithi027.stderr:Export successful 2024-09-13T13:51:45.996 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:45.997 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:45.997 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:45.997 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:45.997 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:45.998 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:45.998+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:46.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:46.099 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:46.099+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:46.153 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:46.153 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:46.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.253494126000007, 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:51:46.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:51:46.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 93.253426269000002, 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.160 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:51:46.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.166 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923451657000001, 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:46.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:46.171 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923417181999994, 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:51:46.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:51:46.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:46.176 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923402195999998, 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.177 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:51:46.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:51:46.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.181 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923388146999997, 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:46.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.186 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923366431999995, 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:51:46.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:51:46.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.191 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.923334586999999, 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:51:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.196 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761637860999997, 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:46.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:51:46.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761604711000004, 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.204 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:46.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.209 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761574492999998, 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:46.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:51:46.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761532144, 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.215 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:51:46.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:51:46.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761497926999994, 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:51:46.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:51:46.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.761464656000001, 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.227 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:51:46.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.232 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 77.359181422999995, 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.233 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:51:46.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.334114814000003, 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.237 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:51:46.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:51:46.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2587 4.c 4:34327e96:::smithi15938319-44:head [write 1772395~538068] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 43.328351859000001, 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012151192999999999, 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2587 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823297+0000", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823897+0000", 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060006600000000001 2024-09-13T13:51:46.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823910+0000", 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2243e-05 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823920+0000", 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0538e-05 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825222+0000", 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001302194 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825304+0000", 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1953999999999995e-05 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825429+0000", 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012415399999999999 2024-09-13T13:51:46.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826790+0000", 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013615050000000001 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835350+0000", 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085598610000000002 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835406+0000", 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5661000000000001e-05 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835447+0000", 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2964 4.c 4:358f784f:::smithi15938319-41:head [rollback 13] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:46.250 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.278046567000001, 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012071405, 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2964 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4699999999999998e-07 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873603+0000", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-06 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873605+0000", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7690000000000001e-06 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873608+0000", 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.615e-06 2024-09-13T13:51:46.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873725+0000", 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116947 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873727+0000", 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4109999999999999e-06 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881339+0000", 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007611835 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881439+0000", 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9967e-05 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881665+0000", 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022542600000000001 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883193+0000", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001528263 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885610+0000", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002417242 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885643+0000", 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2704999999999999e-05 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885673+0000", 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9709e-05 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2965 4.c 4:358f784f:::smithi15938319-41:head [append 0~0] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 29.278036290999999, 2024-09-13T13:51:46.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014521253, 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2965 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:46.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873612+0000", 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2999999999999999e-07 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.259 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873613+0000", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.398e-06 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873616+0000", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7190000000000001e-06 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873618+0000", 2024-09-13T13:51:46.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6170000000000001e-06 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873745+0000", 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126389 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873747+0000", 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4030000000000001e-06 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882806+0000", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090587889999999994 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882861+0000", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4793999999999997e-05 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882955+0000", 2024-09-13T13:51:46.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4377999999999997e-05 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883725+0000", 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076983499999999996 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888075+0000", 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043496630000000001 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888105+0000", 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0735000000000003e-05 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888133+0000", 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7302999999999999e-05 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.264 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3271 4.c 4:34c00adb:::smithi15938319-12:head [write 537890~789585,stat] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 16.215526230999998, 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011376862999999999, 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3271 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:46.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936124+0000", 2024-09-13T13:51:46.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4719999999999998e-06 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937704+0000", 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015798120000000001 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937743+0000", 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8906000000000003e-05 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937754+0000", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0885e-05 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937825+0000", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1216000000000004e-05 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937830+0000", 2024-09-13T13:51:46.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0579999999999997e-06 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940585+0000", 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002755455 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940660+0000", 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940813+0000", 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015292100000000001 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947359+0000", 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065464060000000003 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947381+0000", 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2036999999999999e-05 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947459+0000", 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7338999999999999e-05 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947498+0000", 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9682999999999999e-05 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:46.271 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:46.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:46.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 4.e 2024-09-13T13:51:46.274 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:46.272+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:46.373 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:46.372+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:46.384 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:46.384 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:46.384 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:46.384 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:46.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:46.474 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:46.472+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:46.496 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:46.496 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:46.496 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:46.496 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:46.496 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:46.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:46.574 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:46.573+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:46.612 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 93.692276100000001, 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:51:46.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:51:46.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 93.687476571000005, 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.619 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:51:46.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.624 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 92.838128533000003, 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 92.533328221999994, 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:51:46.631 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:51:46.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.333830933000002, 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:51:46.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:51:46.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:51:46.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.645 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.327970984000004, 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:51:46.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:51:46.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.323026342999995, 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:51:46.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:51:46.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:51:46.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.322815765000001, 2024-09-13T13:51:46.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:46.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:51:46.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:51:46.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:51:46.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.322767666000004, 2024-09-13T13:51:46.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:51:46.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:46.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.239687255000007, 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.676 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:51:46.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.239578543999997, 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:51:46.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:51:46.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.239463263999994, 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:51:46.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:51:46.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.365492161999995, 2024-09-13T13:51:46.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:46.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:51:46.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:46.697 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.365405844999998, 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:51:46.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.702 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.365345474999998, 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:51:46.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:51:46.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:46.707 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.731493329999999, 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:51:46.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:51:46.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2634 4.7 4:e1bd516b:::smithi15938319-19:head [sparse-read 0~2456909] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T13:51:46.714 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.786395314000004, 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087606320000000008, 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2634 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821171+0000", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821172+0000", 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:51:46.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821175+0000", 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5719999999999999e-06 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821180+0000", 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821185+0000", 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6590000000000004e-06 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821322+0000", 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013670500000000001 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.821328+0000", 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8170000000000004e-06 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829522+0000", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081938299999999992 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829600+0000", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7750000000000006e-05 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:05.829932+0000", 2024-09-13T13:51:46.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033246000000000003 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3008 4.7 4:e26a40dd:::smithi15938319-25:head [sparse-read 0~3142009] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.404298882999999, 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0093824860000000006, 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3008 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203272+0000", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6770000000000001e-06 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203280+0000", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203287+0000", 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:51:46.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210677+0000", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073896709999999996 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210743+0000", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6392999999999999e-05 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.212650+0000", 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019070300000000001 2024-09-13T13:51:46.724 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3018 4.7 4:e63fe140:::smithi15938319-1:14 [read 792552~2165480,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.392357639, 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0088409060000000008, 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3018 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215209+0000", 2024-09-13T13:51:46.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215210+0000", 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0299999999999998e-07 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215214+0000", 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4889999999999997e-06 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215228+0000", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3448999999999999e-05 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.215238+0000", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9059999999999997e-06 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.223122+0000", 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0078847910000000004 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.223193+0000", 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1019999999999994e-05 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.224050+0000", 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085654800000000005 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3134 4.7 4:e1bd516b:::smithi15938319-19:head [write 769820~720742,stat] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.434652725999999, 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096203899999999995, 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3134 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172917+0000", 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6759999999999999e-06 2024-09-13T13:51:46.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173729+0000", 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081220499999999998 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173743+0000", 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4073000000000001e-05 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:46.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173749+0000", 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173849+0000", 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9809999999999995e-05 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173917+0000", 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7556000000000002e-05 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.174067+0000", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000150335 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.180026+0000", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059590329999999999 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182455+0000", 2024-09-13T13:51:46.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024288000000000001 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182498+0000", 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2861000000000002e-05 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182534+0000", 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6522e-05 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.735 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:46.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:46.736 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:46.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:46.738 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:46.736+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:46.740 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:46.739+0000 7f002fbdf640 -1 osd.15 126 handle_osd_map injecting map crc failure 2024-09-13T13:51:46.745 INFO:tasks.rados.rados.0.smithi159.stdout:1597: done (2 left) 2024-09-13T13:51:46.745 INFO:tasks.rados.rados.0.smithi159.stdout:1598: delete oid 12 current snap is 43 2024-09-13T13:51:46.745 INFO:tasks.rados.rados.0.smithi159.stdout:1582: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:46.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:51:46.837 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:46.837+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:46.840 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:46.840 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:46.840 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:46.840 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:46.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:46.889 INFO:tasks.rados.rados.0.smithi159.stdout:1592: got (0) Success 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 192 (ObjNum 272 snap 36 seq_num 272) dirty exists 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1582: done (2 left) 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1592: done (1 left) 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1598: done (0 left) 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1599: read oid 27 snap -1 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1599: expect (ObjNum 223 snap 24 seq_num 223) 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1600: read oid 42 snap -1 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1600: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:46.890 INFO:tasks.rados.rados.0.smithi159.stdout:1601: set_chunk oid 19 target oid 2024-09-13T13:51:46.901 INFO:tasks.rados.rados.0.smithi159.stdout:1601: set_chunk oid 19 offset: 1531392 length: 352768 target oid 9a094965a24f051f442e5b9cdabf494a040803571a295c3057557f1fe6d3de24 offset: 0 snap -1 2024-09-13T13:51:46.901 INFO:tasks.rados.rados.0.smithi159.stdout:1602: set_chunk oid 34 target oid 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1602: set_chunk oid 34 offset: 2529792 length: 653824 target oid 52c6e12f6d0e3de098043f8f396914d1811d57e03c8368ec7cb4d0009193a081 offset: 0 snap -1 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1603: set_chunk oid 37 target oid 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1603: set_chunk oid 37 offset: 1616384 length: 260608 target oid smithi15938319- offset: 1616384 snap 34 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1604: read oid 20 snap 38 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1604: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:51:46.916 INFO:tasks.rados.rados.0.smithi159.stdout:1605: write oid 5 current snap is 43 2024-09-13T13:51:46.917 INFO:tasks.rados.rados.0.smithi159.stdout:1605: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-13T13:51:46.922 INFO:tasks.rados.rados.0.smithi159.stdout:1605: writing smithi15938319-5 from 608486 to 1170455 tid 1 2024-09-13T13:51:46.930 INFO:tasks.rados.rados.0.smithi159.stdout:1605: writing smithi15938319-5 from 1871634 to 2505726 tid 2 2024-09-13T13:51:46.933 INFO:tasks.rados.rados.0.smithi159.stdout:1605: writing smithi15938319-5 from 2950640 to 3008486 tid 3 2024-09-13T13:51:46.933 INFO:tasks.rados.rados.0.smithi159.stdout:1606: read oid 46 snap 16 2024-09-13T13:51:46.933 INFO:tasks.rados.rados.0.smithi159.stdout:1606: expect deleted 2024-09-13T13:51:46.933 INFO:tasks.rados.rados.0.smithi159.stdout:1607: set_chunk oid 36 target oid 2024-09-13T13:51:46.938 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:46.937+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1607: set_chunk oid 36 offset: 828928 length: 171008 target oid 150ab7225a0ab0e29821247dcb7f5486c572367ee09ae260d060014154a502db offset: 0 snap -1 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1608: read oid 39 snap -1 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1608: expect (ObjNum 248 snap 28 seq_num 248) 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1609: read oid 2 snap -1 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1609: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:46.939 INFO:tasks.rados.rados.0.smithi159.stdout:1610: set_chunk oid 26 target oid 2024-09-13T13:51:46.955 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:46.955 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:46.955 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:46.957 INFO:tasks.rados.rados.0.smithi159.stdout:1610: set_chunk oid 26 offset: 126464 length: 1953792 target oid e8f4dc9ac3e43c73c06f398c903a06eaffe41a12a8691614232fc5cb000ef8cb offset: 0 snap -1 2024-09-13T13:51:46.957 INFO:tasks.rados.rados.0.smithi159.stdout:1611: read oid 33 snap -1 2024-09-13T13:51:46.958 INFO:tasks.rados.rados.0.smithi159.stdout:1611: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:46.958 INFO:tasks.rados.rados.0.smithi159.stdout:1612: delete oid 13 current snap is 43 2024-09-13T13:51:46.958 INFO:tasks.rados.rados.0.smithi159.stdout:1600: expect (ObjNum 256 snap 30 seq_num 256) 2024-09-13T13:51:46.958 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:46.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:47.039 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:47.039+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.071 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 93.300203898999996, 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.072 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:51:47.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:51:47.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.077 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 93.291173466999993, 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:51:47.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:51:47.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.639167591000003, 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:51:47.082 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:51:47.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:51:47.086 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 88.636133604999998, 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:51:47.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.090 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.760301487000007, 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:51:47.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:51:47.095 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.754394654999999, 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:51:47.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.746959219999994, 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:51:47.100 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:51:47.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.827665478, 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:51:47.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:51:47.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.827596813, 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:47.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:51:47.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.827546698999996, 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.806589337999995, 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.800993134999999, 2024-09-13T13:51:47.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:47.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.799449225000004, 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.799384176999993, 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:51:47.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:51:47.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.192442577999998, 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:51:47.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.152 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 56.336766875000002, 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:51:47.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 56.264871186999997, 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:51:47.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:51:47.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 56.264833666000001, 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.164 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.169 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.151914910999999, 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:51:47.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:51:47.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.176 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.937848502999998, 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:51:47.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:51:47.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:51:47.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.184 INFO:tasks.rados.rados.0.smithi159.stdout:1601: finishing set_chunk to oid 19 2024-09-13T13:51:47.185 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 164 (ObjNum 293 snap 41 seq_num 293) dirty exists 2024-09-13T13:51:47.185 INFO:tasks.rados.rados.0.smithi159.stdout:1599: expect (ObjNum 223 snap 24 seq_num 223) 2024-09-13T13:51:47.185 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.185 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.185 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:47.182+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:47.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:47.198 INFO:tasks.rados.rados.0.smithi159.stdout:1603: finishing set_chunk to oid 37 2024-09-13T13:51:47.198 INFO:tasks.rados.rados.0.smithi159.stdout:1603: got expected ENOENT (src dne) 2024-09-13T13:51:47.198 INFO:tasks.rados.rados.0.smithi159.stdout:1602: finishing set_chunk to oid 34 2024-09-13T13:51:47.198 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 162 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-09-13T13:51:47.198 INFO:tasks.rados.rados.0.smithi159.stdout:1604: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:51:47.285 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:47.285+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:47.288 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.288 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:47.288 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:47.288 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:47.309 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.365 INFO:tasks.rados.rados.0.smithi159.stdout:1605: finishing write tid 1 to smithi15938319-5 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: finishing write tid 2 to smithi15938319-5 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: finishing write tid 3 to smithi15938319-5 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: finishing write tid 4 to smithi15938319-5 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: finishing write tid 5 to smithi15938319-5 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: oid 5 updating version 0 to 197 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: oid 5 updating version 197 to 199 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: oid 5 version 199 is already newer than 198 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: oid 5 version 199 is already newer than 196 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 199 (ObjNum 298 snap 43 seq_num 298) dirty exists 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1605: left oid 5 (ObjNum 298 snap 43 seq_num 298) 2024-09-13T13:51:47.366 INFO:tasks.rados.rados.0.smithi159.stdout:1607: finishing set_chunk to oid 36 2024-09-13T13:51:47.367 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 204 (ObjNum 280 snap 38 seq_num 280) dirty exists 2024-09-13T13:51:47.367 INFO:tasks.rados.rados.0.smithi159.stdout:1608: expect (ObjNum 248 snap 28 seq_num 248) 2024-09-13T13:51:47.379 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.387 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:47.385+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:47.399 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.399 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:47.399 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:47.399 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:47.399 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.403 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:47.428 INFO:tasks.rados.rados.0.smithi159.stdout:1609: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:51:47.488 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:47.487+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:47.495 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:47.510 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 116.275289382, 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:51:47.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:47.515 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 115.267255236, 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:51:47.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.871557077000006, 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:51:47.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.324653374999997, 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:51:47.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:51:47.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.319812550999998, 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.529 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:51:47.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 34.319099575999999, 2024-09-13T13:51:47.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:47.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.538 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.538 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:47.576 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1522501 2024-09-13T13:51:47.588 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:47.588+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:47.627 INFO:tasks.rados.rados.0.smithi159.stdout:1610: finishing set_chunk to oid 26 2024-09-13T13:51:47.627 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 126464 length: 1953792 2024-09-13T13:51:47.627 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 258 snap 30 seq_num 258) dirty exists 2024-09-13T13:51:47.627 INFO:tasks.rados.rados.0.smithi159.stdout:1611: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:47.630 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1861071 2024-09-13T13:51:47.641 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.641 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:47.641 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:47.642 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.644 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.650 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:47.674 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:47.689 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:47.687+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:47.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1599: done (13 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1600: done (12 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1601: done (11 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1602: done (10 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1603: done (9 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1604: done (8 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1605: done (7 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1606: done (6 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1607: done (5 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1608: done (4 left) 2024-09-13T13:51:47.704 INFO:tasks.rados.rados.0.smithi159.stdout:1609: done (3 left) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1610: done (2 left) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1611: done (1 left) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1612: done (0 left) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1613: read oid 39 snap 40 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1613: expect (ObjNum 248 snap 28 seq_num 248) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1614: rollback oid 22 current snap is 43 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 22 to 38 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1615: rollback oid 9 current snap is 43 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 9 to 38 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1616: read oid 41 snap -1 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1616: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:47.705 INFO:tasks.rados.rados.0.smithi159.stdout:1617: delete oid 13 current snap is 43 2024-09-13T13:51:47.706 INFO:tasks.rados.rados.0.smithi159.stdout:1617: done (4 left) 2024-09-13T13:51:47.706 INFO:tasks.rados.rados.0.smithi159.stdout:1618: set_chunk oid 25 target oid 2024-09-13T13:51:47.716 INFO:teuthology.orchestra.run.smithi027.stdout: marking collection for removal 2024-09-13T13:51:47.716 INFO:teuthology.orchestra.run.smithi027.stdout:setting '_remove' omap key 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi027.stdout:finish_remove_pgs 4.e_head removing 4.e 2024-09-13T13:51:47.721 INFO:tasks.rados.rados.0.smithi159.stdout:1618: set_chunk oid 25 offset: 240640 length: 1212416 target oid f6f4d442b842fe14da34f1e0143b9dae39d3905559956bafe1d4c6a3e7f48d5b offset: 0 snap -1 2024-09-13T13:51:47.721 INFO:tasks.rados.rados.0.smithi159.stdout:1619: read oid 40 snap -1 2024-09-13T13:51:47.721 INFO:tasks.rados.rados.0.smithi159.stdout:1619: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:47.721 INFO:tasks.rados.rados.0.smithi159.stdout:1620: write (excl) oid 30 current snap is 43 2024-09-13T13:51:47.721 INFO:tasks.rados.rados.0.smithi159.stdout:1620: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-13T13:51:47.725 INFO:teuthology.orchestra.run.smithi027.stdout:Remove successful 2024-09-13T13:51:47.728 INFO:tasks.rados.rados.0.smithi159.stdout:1620: writing smithi15938319-30 from 658907 to 1446623 tid 1 2024-09-13T13:51:47.734 INFO:tasks.rados.rados.0.smithi159.stdout:1620: writing smithi15938319-30 from 1930727 to 2428499 tid 2 2024-09-13T13:51:47.737 INFO:tasks.rados.rados.0.smithi159.stdout:1620: writing smithi15938319-30 from 3032909 to 3058907 tid 3 2024-09-13T13:51:47.737 INFO:tasks.rados.rados.0.smithi159.stdout:1621: snap_remove snap 41 2024-09-13T13:51:47.789 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:47.788+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:47.795 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.795 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:47.795 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:51:47.795 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:51:47.795 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:47.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:47.835 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:47.836 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:47.857 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:47.857 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:47.879 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:47.879 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:47.890 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:47.890+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:47.900 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:47.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:47.901 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:47.908 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:47.908 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:51:47.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.139899510000006, 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:51:47.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:51:47.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.138922312000005, 2024-09-13T13:51:47.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:51:47.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.137167646999998, 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.922 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:51:47.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:51:47.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.136775434, 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.928 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:51:47.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:51:47.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.136738106999999, 2024-09-13T13:51:47.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:47.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:51:47.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.131939754000001, 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.941 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:47.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:51:47.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.116304529000004, 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:51:47.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.116198178999994, 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:51:47.953 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:51:47.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.959 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.108264409, 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.960 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:51:47.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:51:47.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:51:47.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:47.966 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.089875851000002, 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.967 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:51:47.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:51:47.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.089749061000006, 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:47.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:51:47.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.008550638000003, 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:51:47.979 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:51:47.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.985 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 95.008449186999997, 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:51:47.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:51:47.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:51:47.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.447096420999998, 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:47.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:51:47.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:51:47.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:47.998 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.441244878000006, 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:47.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:51:48.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.002 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 79.164435241999996, 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:48.003 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.007 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 74.179367427000003, 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.008 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:51:48.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:51:48.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 73.173346151999993, 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:48.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.7.23:20 3.1 3:8dbd923e:::ae8b86d07d375a083683440b6bda0e0f67b5195d585764d7ac407424ac114e59:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.8899109789999997, 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89571672999999996, 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016793+0000", 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6600000000000005e-07 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016800+0000", 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1199999999999996e-06 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910876+0000", 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89407579400000003 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910884+0000", 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910982+0000", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7763999999999994e-05 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911101+0000", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119942 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911338+0000", 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023690000000000001 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912020+0000", 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068117299999999998 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912443+0000", 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042320999999999998 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912478+0000", 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4768999999999999e-05 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912509+0000", 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1226999999999998e-05 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(3.1 126/125 [PushReplyOp(3:ac2af9f5:::e6b4f9f8265bfe5a786cb2bd77c53a79b3e484041c7c0186a813c80a2460efe6:head)])", 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.8604921679999999, 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208631544, 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9399999999999995e-07 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046216+0000", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3959999999999999e-06 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254500+0000", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208283991 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254505+0000", 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.789e-06 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254594+0000", 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8101000000000002e-05 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254842+0000", 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024877299999999998 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:48.033 INFO:tasks.rados.rados.0.smithi159.stdout:1621: done (7 left) 2024-09-13T13:51:48.033 INFO:tasks.rados.rados.0.smithi159.stdout:1622: tier_evict oid 5 2024-09-13T13:51:48.033 INFO:tasks.rados.rados.0.smithi159.stdout:1622: tier_evict oid 5 snap -1 2024-09-13T13:51:48.033 INFO:tasks.rados.rados.0.smithi159.stdout:1614: finishing rollback tid 0 to smithi15938319-22 2024-09-13T13:51:48.033 INFO:tasks.rados.rados.0.smithi159.stdout:1615: finishing rollback tid 0 to smithi15938319-9 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:1614: finishing rollback tid 1 to smithi15938319-22 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:1615: finishing rollback tid 1 to smithi15938319-9 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:1614: finishing rollback tid 2 to smithi15938319-22 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 42 (ObjNum 277 snap 38 seq_num 277) dirty exists 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:1615: finishing rollback tid 2 to smithi15938319-9 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 249 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:51:48.034 INFO:tasks.rados.rados.0.smithi159.stdout:1613: expect (ObjNum 248 snap 28 seq_num 248) 2024-09-13T13:51:48.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:48.034 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:48.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:48.038 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:48.030+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:48.052 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:48.052 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:48.053 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:48.058 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:48.057+0000 7ffaeece8640 -1 osd.0 127 handle_osd_map injecting map crc failure 2024-09-13T13:51:48.073 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:48.073 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:48.078 INFO:tasks.rados.rados.0.smithi159.stdout:1616: expect (ObjNum 255 snap 30 seq_num 255) 2024-09-13T13:51:48.078 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:48.094 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:48.095 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:48.115 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:48.116 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:48.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:48.121+0000 7f750dd9b640 0 -- 172.21.15.27:0/3842524903 >> v2:172.21.15.159:6805/4073091144 conn(0x7f7510222460 msgr2=0x7f7510224850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:48.132 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:48.130+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:48.135 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:48.134+0000 7fe0f9e03640 -1 osd.2 127 handle_osd_map injecting map crc failure 2024-09-13T13:51:48.142 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:48.142 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:48.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:48.146 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.147 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.155 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:48.166 INFO:tasks.rados.rados.0.smithi159.stdout:1618: finishing set_chunk to oid 25 2024-09-13T13:51:48.166 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 240640 length: 1212416 2024-09-13T13:51:48.166 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 0 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:51:48.166 INFO:tasks.rados.rados.0.smithi159.stdout:1619: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:48.193 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2297174 2024-09-13T13:51:48.207 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 47187 2024-09-13T13:51:48.233 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:48.231+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:48.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.e.4 2024-09-13T13:51:48.259 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.259 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.260 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:48.268 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:48.314 INFO:tasks.rados.rados.0.smithi159.stdout:1620: finishing write tid 1 to smithi15938319-30 2024-09-13T13:51:48.314 INFO:tasks.rados.rados.0.smithi159.stdout:1620: finishing write tid 2 to smithi15938319-30 2024-09-13T13:51:48.314 INFO:tasks.rados.rados.0.smithi159.stdout:1620: finishing write tid 3 to smithi15938319-30 2024-09-13T13:51:48.314 INFO:tasks.rados.rados.0.smithi159.stdout:1620: finishing write tid 4 to smithi15938319-30 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: finishing write tid 5 to smithi15938319-30 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: oid 30 updating version 0 to 269 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: oid 30 version 269 is already newer than 267 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: oid 30 version 269 is already newer than 268 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: oid 30 version 269 is already newer than 266 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 269 (ObjNum 301 snap 43 seq_num 301) dirty exists 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1620: left oid 30 (ObjNum 301 snap 43 seq_num 301) 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1622: got (22) Invalid argument 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1613: done (7 left) 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1614: done (6 left) 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1615: done (5 left) 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1616: done (4 left) 2024-09-13T13:51:48.315 INFO:tasks.rados.rados.0.smithi159.stdout:1618: done (3 left) 2024-09-13T13:51:48.316 INFO:tasks.rados.rados.0.smithi159.stdout:1619: done (2 left) 2024-09-13T13:51:48.316 INFO:tasks.rados.rados.0.smithi159.stdout:1620: done (1 left) 2024-09-13T13:51:48.316 INFO:tasks.rados.rados.0.smithi159.stdout:1622: done (0 left) 2024-09-13T13:51:48.316 INFO:tasks.rados.rados.0.smithi159.stdout:1623: write oid 10 current snap is 43 2024-09-13T13:51:48.316 INFO:tasks.rados.rados.0.smithi159.stdout:1623: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-13T13:51:48.320 INFO:tasks.rados.rados.0.smithi159.stdout:1623: writing smithi15938319-10 from 675714 to 1138679 tid 1 2024-09-13T13:51:48.325 INFO:tasks.rados.rados.0.smithi159.stdout:1623: writing smithi15938319-10 from 1589209 to 2230639 tid 2 2024-09-13T13:51:48.330 INFO:tasks.rados.rados.0.smithi159.stdout:1623: writing smithi15938319-10 from 2943979 to 3075714 tid 3 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1624: read oid 33 snap -1 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1624: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1625: tier_evict oid 47 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1625: tier_evict oid 47 snap -1 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1623: finishing write tid 1 to smithi15938319-10 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1623: finishing write tid 2 to smithi15938319-10 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1626: tier_evict oid 48 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1626: tier_evict oid 48 snap -1 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1627: read oid 7 snap -1 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1627: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:51:48.331 INFO:tasks.rados.rados.0.smithi159.stdout:1628: snap_remove snap 36 2024-09-13T13:51:48.333 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:48.332+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:48.373 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.373 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:48.435 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:48.433+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:48.485 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.485 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.493 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:48.535 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:48.534+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:48.595 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:48.648 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:48.647+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:48.737 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:48.737+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:48.747 INFO:tasks.rados.rados.0.smithi159.stdout:1628: done (5 left) 2024-09-13T13:51:48.748 INFO:tasks.rados.rados.0.smithi159.stdout:1629: set_chunk oid 46 target oid 2024-09-13T13:51:48.750 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:48.749+0000 7f2282f0c640 -1 osd.10 128 handle_osd_map injecting map crc failure 2024-09-13T13:51:48.762 INFO:tasks.rados.rados.0.smithi159.stdout:1629: set_chunk oid 46 offset: 848384 length: 981504 target oid f34b51f0551f8a0cc11131c7c5e7bf706073c85fdd63cb1aaa62262995ca818a offset: 0 snap -1 2024-09-13T13:51:48.762 INFO:tasks.rados.rados.0.smithi159.stdout:1630: write (excl) oid 38 current snap is 43 2024-09-13T13:51:48.762 INFO:tasks.rados.rados.0.smithi159.stdout:1630: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-13T13:51:48.769 INFO:tasks.rados.rados.0.smithi159.stdout:1630: writing smithi15938319-38 from 692521 to 1230735 tid 1 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1630: writing smithi15938319-38 from 1931338 to 2716426 tid 2 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1631: read oid 26 snap -1 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1631: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1632: read oid 25 snap 28 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1632: expect (ObjNum 230 snap 25 seq_num 230) 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1633: tier_evict oid 11 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1633: tier_evict oid 11 snap -1 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1625: got (22) Invalid argument 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1625: done (9 left) 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1634: tier_flush oid 39 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1634: tier_flush oid 39 snap -1 2024-09-13T13:51:48.778 INFO:tasks.rados.rados.0.smithi159.stdout:1626: got (22) Invalid argument 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: finishing write tid 3 to smithi15938319-10 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: finishing write tid 4 to smithi15938319-10 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: finishing write tid 5 to smithi15938319-10 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: oid 10 updating version 0 to 307 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: oid 10 updating version 307 to 309 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: oid 10 updating version 309 to 310 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: oid 10 version 310 is already newer than 308 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 10 v 310 (ObjNum 302 snap 43 seq_num 302) dirty exists 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1623: left oid 10 (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:51:48.779 INFO:tasks.rados.rados.0.smithi159.stdout:1624: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:48.838 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:48.837+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:48.839 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 293484 2024-09-13T13:51:48.861 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.861 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.867 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:48.867 INFO:tasks.rados.rados.0.smithi159.stdout:1627: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:51:48.872 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 668246 2024-09-13T13:51:48.938 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:48.938+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:48.969 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:48.969 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:48.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:49.039 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:49.038+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:1629: finishing set_chunk to oid 46 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 848384 length: 981504 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 0 (ObjNum 247 snap 28 seq_num 247) dirty exists 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:1630: finishing write tid 1 to smithi15938319-38 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:1623: done (9 left) 2024-09-13T13:51:49.108 INFO:tasks.rados.rados.0.smithi159.stdout:1624: done (8 left) 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1626: done (7 left) 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1627: done (6 left) 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1629: done (5 left) 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1635: read oid 43 snap -1 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1635: expect (ObjNum 290 snap 38 seq_num 290) 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1633: got (22) Invalid argument 2024-09-13T13:51:49.109 INFO:tasks.rados.rados.0.smithi159.stdout:1631: expect (ObjNum 258 snap 30 seq_num 258) 2024-09-13T13:51:49.134 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:49.174 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:49.173+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:49.206 INFO:tasks.rados.rados.0.smithi159.stdout:1630: finishing write tid 2 to smithi15938319-38 2024-09-13T13:51:49.206 INFO:tasks.rados.rados.0.smithi159.stdout:1630: finishing write tid 3 to smithi15938319-38 2024-09-13T13:51:49.206 INFO:tasks.rados.rados.0.smithi159.stdout:1632: expect (ObjNum 230 snap 25 seq_num 230) 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.241 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:49.240+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:49.242 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:49.288 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3677602 2024-09-13T13:51:49.342 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:49.341+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:49.442 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:49.442+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1634: got (0) Success 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 248 snap 28 seq_num 248) dirty exists 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1636: read oid 29 snap -1 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1636: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1631: done (6 left) 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1632: done (5 left) 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1633: done (4 left) 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1634: done (3 left) 2024-09-13T13:51:49.499 INFO:tasks.rados.rados.0.smithi159.stdout:1637: read oid 13 snap -1 2024-09-13T13:51:49.500 INFO:tasks.rados.rados.0.smithi159.stdout:1637: expect deleted 2024-09-13T13:51:49.500 INFO:tasks.rados.rados.0.smithi159.stdout:1638: set_chunk oid 30 target oid 2024-09-13T13:51:49.500 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.500 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.501 INFO:tasks.rados.rados.0.smithi159.stdout:1638: set_chunk oid 30 offset: 400896 length: 71168 target oid 8f293a2c394d3135faefcccdc9a37de80ab038eead17d96fc88ac67afbdf0d88 offset: 0 snap -1 2024-09-13T13:51:49.501 INFO:tasks.rados.rados.0.smithi159.stdout:1639: write oid 22 current snap is 43 2024-09-13T13:51:49.501 INFO:tasks.rados.rados.0.smithi159.stdout:1639: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-13T13:51:49.508 INFO:tasks.rados.rados.0.smithi159.stdout:1639: writing smithi15938319-22 from 709328 to 1322791 tid 1 2024-09-13T13:51:49.509 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:49.515 INFO:tasks.rados.rados.0.smithi159.stdout:1639: writing smithi15938319-22 from 1989820 to 2634919 tid 2 2024-09-13T13:51:49.515 INFO:tasks.rados.rados.0.smithi159.stdout:1640: read oid 47 snap 38 2024-09-13T13:51:49.516 INFO:tasks.rados.rados.0.smithi159.stdout:1640: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:51:49.516 INFO:tasks.rados.rados.0.smithi159.stdout:1641: write (excl) oid 21 current snap is 43 2024-09-13T13:51:49.516 INFO:tasks.rados.rados.0.smithi159.stdout:1641: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-13T13:51:49.522 INFO:tasks.rados.rados.0.smithi159.stdout:1641: writing smithi15938319-21 from 726135 to 1131200 tid 1 2024-09-13T13:51:49.530 INFO:tasks.rados.rados.0.smithi159.stdout:1641: writing smithi15938319-21 from 1764655 to 2553412 tid 2 2024-09-13T13:51:49.533 INFO:tasks.rados.rados.0.smithi159.stdout:1641: writing smithi15938319-21 from 3026248 to 3126135 tid 3 2024-09-13T13:51:49.533 INFO:tasks.rados.rados.0.smithi159.stdout:1642: write oid 23 current snap is 43 2024-09-13T13:51:49.533 INFO:tasks.rados.rados.0.smithi159.stdout:1635: expect (ObjNum 290 snap 38 seq_num 290) 2024-09-13T13:51:49.534 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:49.543 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:49.542+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:49.614 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.614 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.624 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:49.644 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:49.643+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:49.745 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:49.745+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:49.748 INFO:tasks.rados.rados.0.smithi159.stdout:1638: finishing set_chunk to oid 30 2024-09-13T13:51:49.749 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 269 (ObjNum 301 snap 43 seq_num 301) dirty exists 2024-09-13T13:51:49.749 INFO:tasks.rados.rados.0.smithi159.stdout:1636: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:51:49.769 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.769 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.777 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:49.846 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:49.846+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi027.stdout:get_pg_num_history pg_num_history pg_num_history(e123 pg_nums {1={23=1},2={25=8},3={33=4},4={34=16}} deleted_pools ) 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi027.stdout:Importing pgid 4.e 2024-09-13T13:51:49.892 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:709ad093:::smithi15938319-50:12# 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1639: finishing write tid 1 to smithi15938319-22 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1630: finishing write tid 4 to smithi15938319-38 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1630: oid 38 updating version 0 to 206 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1630: oid 38 updating version 206 to 208 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1630: oid 38 version 208 is already newer than 207 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 208 (ObjNum 303 snap 43 seq_num 303) dirty exists 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1630: left oid 38 (ObjNum 303 snap 43 seq_num 303) 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1639: finishing write tid 2 to smithi15938319-22 2024-09-13T13:51:49.894 INFO:tasks.rados.rados.0.smithi159.stdout:1642: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:709ad093:::smithi15938319-50:head# 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi027.stdout:snapset 22=[]:{12=[12]} 2024-09-13T13:51:49.900 INFO:tasks.rados.rados.0.smithi159.stdout:1642: writing smithi15938319-23 from 742942 to 1223256 tid 1 2024-09-13T13:51:49.901 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:70cd1372:::smithi15938319-9:1f# 2024-09-13T13:51:49.904 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:70cd1372:::smithi15938319-9:head# 2024-09-13T13:51:49.904 INFO:teuthology.orchestra.run.smithi027.stdout:snapset 1f=[]:{1f=[1e,1a,17,14,12]} 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1642: writing smithi15938319-23 from 1823137 to 2471905 tid 2 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1630: done (8 left) 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1635: done (7 left) 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1636: done (6 left) 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1637: done (5 left) 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1638: done (4 left) 2024-09-13T13:51:49.908 INFO:tasks.rados.rados.0.smithi159.stdout:1643: read oid 48 snap -1 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1643: expect (ObjNum 296 snap 41 seq_num 296) 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1644: read oid 44 snap -1 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1644: expect (ObjNum 210 snap 22 seq_num 210) 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1645: read oid 16 snap 34 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1645: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1646: delete oid 32 current snap is 43 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1639: finishing write tid 3 to smithi15938319-22 2024-09-13T13:51:49.909 INFO:tasks.rados.rados.0.smithi159.stdout:1639: finishing write tid 4 to smithi15938319-22 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1639: oid 22 updating version 0 to 48 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1639: oid 22 updating version 48 to 49 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1639: oid 22 version 49 is already newer than 47 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 49 (ObjNum 304 snap 43 seq_num 304) dirty exists 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1639: left oid 22 (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: finishing write tid 1 to smithi15938319-21 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: finishing write tid 2 to smithi15938319-21 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: finishing write tid 3 to smithi15938319-21 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: finishing write tid 4 to smithi15938319-21 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: finishing write tid 5 to smithi15938319-21 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: oid 21 updating version 0 to 144 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: oid 21 updating version 144 to 147 2024-09-13T13:51:49.910 INFO:tasks.rados.rados.0.smithi159.stdout:1641: oid 21 version 147 is already newer than 146 2024-09-13T13:51:49.911 INFO:tasks.rados.rados.0.smithi159.stdout:1641: oid 21 version 147 is already newer than 145 2024-09-13T13:51:49.911 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 147 (ObjNum 305 snap 43 seq_num 305) dirty exists 2024-09-13T13:51:49.911 INFO:tasks.rados.rados.0.smithi159.stdout:1641: left oid 21 (ObjNum 305 snap 43 seq_num 305) 2024-09-13T13:51:49.911 INFO:tasks.rados.rados.0.smithi159.stdout:1640: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:51:49.911 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:717ade62:::smithi15938319-24:13# 2024-09-13T13:51:49.912 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:717ade62:::smithi15938319-24:1b# 2024-09-13T13:51:49.916 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:717ade62:::smithi15938319-24:head# 2024-09-13T13:51:49.916 INFO:teuthology.orchestra.run.smithi027.stdout:snapset 1d=[]:{13=[12],1b=[1a,17,14]} 2024-09-13T13:51:49.918 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:721f6cf4:::smithi15938319-18:13# 2024-09-13T13:51:49.921 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:721f6cf4:::smithi15938319-18:14# 2024-09-13T13:51:49.924 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:721f6cf4:::smithi15938319-18:1d# 2024-09-13T13:51:49.927 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:721f6cf4:::smithi15938319-18:21# 2024-09-13T13:51:49.930 INFO:teuthology.orchestra.run.smithi027.stdout:Write #4:721f6cf4:::smithi15938319-18:head# 2024-09-13T13:51:49.930 INFO:teuthology.orchestra.run.smithi027.stdout:snapset 21=[]:{13=[12],14=[14],1d=[1a,17],21=[1e]} 2024-09-13T13:51:49.933 INFO:teuthology.orchestra.run.smithi027.stdout:write_pg epoch 120 info 4.e( v 120'239 (0'0,120'239] local-lis/les=34/35 n=12 ec=34/34 lis/c=34/34 les/c/f=35/35/0 sis=34) 2024-09-13T13:51:49.934 INFO:teuthology.orchestra.run.smithi027.stdout:Import successful 2024-09-13T13:51:49.947 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:49.946+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:49.998 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:49.998 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:50.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:50.019 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1888120 2024-09-13T13:51:50.030 INFO:tasks.rados.rados.0.smithi159.stdout:1642: finishing write tid 1 to smithi15938319-23 2024-09-13T13:51:50.030 INFO:tasks.rados.rados.0.smithi159.stdout:1642: finishing write tid 2 to smithi15938319-23 2024-09-13T13:51:50.030 INFO:tasks.rados.rados.0.smithi159.stdout:1642: finishing write tid 3 to smithi15938319-23 2024-09-13T13:51:50.030 INFO:tasks.rados.rados.0.smithi159.stdout:1642: finishing write tid 4 to smithi15938319-23 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1642: oid 23 updating version 0 to 229 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1642: oid 23 version 229 is already newer than 227 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1642: oid 23 version 229 is already newer than 228 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 229 (ObjNum 306 snap 43 seq_num 306) dirty exists 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1642: left oid 23 (ObjNum 306 snap 43 seq_num 306) 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1639: done (7 left) 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1640: done (6 left) 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1641: done (5 left) 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1642: done (4 left) 2024-09-13T13:51:50.031 INFO:tasks.rados.rados.0.smithi159.stdout:1646: done (3 left) 2024-09-13T13:51:50.032 INFO:tasks.rados.rados.0.smithi159.stdout:1647: set_chunk oid 32 target oid 2024-09-13T13:51:50.032 INFO:tasks.rados.rados.0.smithi159.stdout:1647: set_chunk oid 32 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:51:50.032 INFO:tasks.rados.rados.0.smithi159.stdout:1648: snap_remove snap 16 2024-09-13T13:51:50.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:51:50.048 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:50.046+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:50.121 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:50.121 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:50.130 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:50.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:50.149 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:50.148+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:50.249 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:50.249+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:50.273 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:50.273 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:50.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:50.283 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:50.283 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:50.305 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:50.305 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:50.326 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:50.327 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:50.347 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:50.351 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:50.349+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:50.369 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:50.369 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:50.386 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:50.386 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:50.391 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:50.395 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:50.414 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:50.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:50.435 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:50.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /var/log/ceph/exp.4.e.4 2024-09-13T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:50.459 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:50.480 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:50.478+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:50.480 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:50.481 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:50.482 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-09-13T13:51:50.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:51:50.483 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:50.504 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:50.505 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:50.524 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:50.524 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:50.540 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:50.540 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:50.540 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:50.546 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:50.546 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:50.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:50.552 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:50.552+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:50.567 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:50.568 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:50.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:50.572+0000 7f67a7fff640 0 -- 172.21.15.27:0/3013068027 >> v2:172.21.15.159:6805/4073091144 conn(0x7f67a8222460 msgr2=0x7f67a8224850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:50.594 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:50.594 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:50.597 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:50.653 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:50.652+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:50.694 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 98.431517912000004, 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.695 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.696 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:51:50.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:51:50.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 98.430564412999999, 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:51:50.700 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.701 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:51:50.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:51:50.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:51:50.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.708 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:51:50.709 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 98.204314439000001, 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:51:50.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:51:50.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:51:50.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:51:50.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 86.919728152999994, 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.718 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.721 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:51:50.722 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 82.474362373000005, 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.723 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:51:50.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:51:50.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.727 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.366817580000003, 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:50.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.366068691000002, 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:50.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:51:50.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.947734509, 2024-09-13T13:51:50.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:50.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:50.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.942606151, 2024-09-13T13:51:50.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:50.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:51:50.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:51:50.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:51:50.747 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 59.941694595000001, 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:50.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:51:50.752 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 53.913608506999999, 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:51:50.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:138 4.8 e80/34 4:1b24d731:::smithi15938319-13:11 v 80'75, mlcod=78'69)", 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 53.912942182999998, 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038853730000000001, 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.757 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780312+0000", 2024-09-13T13:51:50.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780322+0000", 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0383999999999999e-05 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780367+0000", 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4783000000000001e-05 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780378+0000", 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0292000000000001e-05 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780701+0000", 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032313399999999999 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780722+0000", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1353e-05 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784153+0000", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003430986 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784196+0000", 2024-09-13T13:51:50.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3498999999999998e-05 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2559 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'82, mlcod=84'79)", 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.847708212000001, 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040899279999999996, 2024-09-13T13:51:50.762 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845546+0000", 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845704+0000", 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015839000000000001 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845732+0000", 2024-09-13T13:51:50.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7736e-05 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845738+0000", 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845791+0000", 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845807+0000", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5755000000000001e-05 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849592+0000", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037853499999999998 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849635+0000", 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2669000000000001e-05 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.767 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.844334162999999, 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:51:50.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:51:50.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 48.843566115000002, 2024-09-13T13:51:50.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:51:50.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:51:50.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.487252835, 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:51:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.787 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3424 4.8 e118/34 4:1b24d731:::smithi15938319-13:head v 118'122, mlcod=115'119)", 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.710645611, 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041392160000000002, 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.788 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.34e-07 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983037+0000", 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042880899999999999 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983075+0000", 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7994000000000002e-05 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983083+0000", 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9030000000000004e-06 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983136+0000", 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3353999999999998e-05 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983152+0000", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5583999999999999e-05 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986717+0000", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035656059999999998 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986747+0000", 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9532000000000001e-05 2024-09-13T13:51:50.792 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.279845569999999, 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.793 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:51:50.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.387160591000001, 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:51:50.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:51:50.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:50.802 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.7355689399999998, 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:50.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:51:50.806 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.807 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.807 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:50.807 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:50.807 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:50.808 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:50.809 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:50.807+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:50.810 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:51:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:51:50.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-13T13:51:50.385764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398226183,"num_objects":259,"num_object_clones":106,"num_object_copies":518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":4,"num_read":4067,"num_read_kb":3016577,"num_write":2049,"num_write_kb":577493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":317789653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3242,"ondisk_log_size":3242,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":1052864,"kb_used_data":618384,"kb_used_omap":23,"kb_used_meta":434216,"kb_avail":1414524736,"statfs":{"total":1449551462400,"available":1448473329664,"internally_reserved":0,"allocated":633225216,"data_stored":782074609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23924,"internal_metadata":444637836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15405832,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-1,"num_read":132,"num_read_kb":126768,"num_write":43,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":24882250,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.645732"},"pg_stats":[{"pgid":"4.f","version":"126'209","reported_seq":662,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.960886+0000","last_change":"2024-09-13T13:51:39.887455+0000","last_active":"2024-09-13T13:51:45.960886+0000","last_peered":"2024-09-13T13:51:45.960886+0000","last_clean":"2024-09-13T13:51:45.960886+0000","last_became_active":"2024-09-13T13:50:06.073131+0000","last_became_peered":"2024-09-13T13:50:06.073131+0000","last_unstale":"2024-09-13T13:51:45.960886+0000","last_undegraded":"2024-09-13T13:51:45.960886+0000","last_fullsized":"2024-09-13T13:51:45.960886+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'188","last_scrub_stamp":"2024-09-13T13:51:25.039858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.039858+0000","objects_scrubbed":8,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:49.708910+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0046137000000000001,"stat_sum":{"num_bytes":13877479,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":251,"num_read_kb":196701,"num_write":93,"num_write_kb":22889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"4.e","version":"120'239","reported_seq":786,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.856435+0000","last_change":"2024-09-13T13:51:45.855953+0000","last_active":"2024-09-13T13:51:45.856435+0000","last_peered":"2024-09-13T13:51:45.856435+0000","last_clean":"2024-09-13T13:51:45.856435+0000","last_became_active":"2024-09-13T13:51:45.749931+0000","last_became_peered":"2024-09-13T13:51:45.749931+0000","last_unstale":"2024-09-13T13:51:45.856435+0000","last_undegraded":"2024-09-13T13:51:45.856435+0000","last_fullsized":"2024-09-13T13:51:45.856435+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'178","last_scrub_stamp":"2024-09-13T13:51:13.109829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.109829+0000","objects_scrubbed":9,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:14.702922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0065814630000000001,"stat_sum":{"num_bytes":21215512,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":394,"num_read_kb":308562,"num_write":114,"num_write_kb":29909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":14756273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"4.d","version":"120'53","reported_seq":460,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:44.771793+0000","last_change":"2024-09-13T13:51:39.882891+0000","last_active":"2024-09-13T13:51:44.771793+0000","last_peered":"2024-09-13T13:51:44.771793+0000","last_clean":"2024-09-13T13:51:44.771793+0000","last_became_active":"2024-09-13T13:50:06.075524+0000","last_became_peered":"2024-09-13T13:50:06.075524+0000","last_unstale":"2024-09-13T13:51:44.771793+0000","last_undegraded":"2024-09-13T13:51:44.771793+0000","last_fullsized":"2024-09-13T13:51:44.771793+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'38","last_scrub_stamp":"2024-09-13T13:51:21.332048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:21.332048+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:30.340468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002143196,"stat_sum":{"num_bytes":6671280,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":71,"num_read_kb":60736,"num_write":25,"num_write_kb":6222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"4.c","version":"120'213","reported_seq":729,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:44.777063+0000","last_change":"2024-09-13T13:51:39.884778+0000","last_active":"2024-09-13T13:51:44.777063+0000","last_peered":"2024-09-13T13:51:44.777063+0000","last_clean":"2024-09-13T13:51:44.777063+0000","last_became_active":"2024-09-13T13:50:28.729220+0000","last_became_peered":"2024-09-13T13:50:28.729220+0000","last_unstale":"2024-09-13T13:51:44.777063+0000","last_undegraded":"2024-09-13T13:51:44.777063+0000","last_fullsized":"2024-09-13T13:51:44.777063+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'162","last_scrub_stamp":"2024-09-13T13:51:12.406192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:12.406192+0000","objects_scrubbed":8,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:12.865218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0040316759999999997,"stat_sum":{"num_bytes":22292535,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":311,"num_read_kb":279735,"num_write":106,"num_write_kb":26915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11131276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"4.b","version":"127'269","reported_seq":719,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006828+0000","last_change":"2024-09-13T13:51:48.007022+0000","last_active":"2024-09-13T13:51:48.006828+0000","last_peered":"2024-09-13T13:51:48.006828+0000","last_clean":"2024-09-13T13:51:48.006828+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:51:48.006828+0000","last_undegraded":"2024-09-13T13:51:48.006828+0000","last_fullsized":"2024-09-13T13:51:48.006828+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'224","last_scrub_stamp":"2024-09-13T13:51:25.879055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.879055+0000","objects_scrubbed":16,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:52.165940+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00036324399999999998,"stat_sum":{"num_bytes":34258690,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":223,"num_read_kb":154035,"num_write":158,"num_write_kb":43790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.a","version":"128'201","reported_seq":779,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.005887+0000","last_change":"2024-09-13T13:51:48.006066+0000","last_active":"2024-09-13T13:51:48.005887+0000","last_peered":"2024-09-13T13:51:48.005887+0000","last_clean":"2024-09-13T13:51:48.005887+0000","last_became_active":"2024-09-13T13:51:02.821659+0000","last_became_peered":"2024-09-13T13:51:02.821659+0000","last_unstale":"2024-09-13T13:51:48.005887+0000","last_undegraded":"2024-09-13T13:51:48.005887+0000","last_fullsized":"2024-09-13T13:51:48.005887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'158","last_scrub_stamp":"2024-09-13T13:51:24.201568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:24.201568+0000","objects_scrubbed":11,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:49.479666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019803059999999998,"stat_sum":{"num_bytes":26967303,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":337,"num_read_kb":330466,"num_write":93,"num_write_kb":27234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":34232380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.9","version":"127'42","reported_seq":434,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:47.707070+0000","last_change":"2024-09-13T13:51:46.749938+0000","last_active":"2024-09-13T13:51:47.707070+0000","last_peered":"2024-09-13T13:51:47.707070+0000","last_clean":"2024-09-13T13:51:47.707070+0000","last_became_active":"2024-09-13T13:50:12.197810+0000","last_became_peered":"2024-09-13T13:50:12.197810+0000","last_unstale":"2024-09-13T13:51:47.707070+0000","last_undegraded":"2024-09-13T13:51:47.707070+0000","last_fullsized":"2024-09-13T13:51:47.707070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'21","last_scrub_stamp":"2024-09-13T13:51:21.631083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:21.631083+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:28.265320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00297263,"stat_sum":{"num_bytes":3783473,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":51272,"num_write":22,"num_write_kb":4780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5],"acting":[15,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"4.8","version":"120'130","reported_seq":547,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.160719+0000","last_change":"2024-09-13T13:51:39.882360+0000","last_active":"2024-09-13T13:51:45.160719+0000","last_peered":"2024-09-13T13:51:45.160719+0000","last_clean":"2024-09-13T13:51:45.160719+0000","last_became_active":"2024-09-13T13:50:06.075916+0000","last_became_peered":"2024-09-13T13:50:06.075916+0000","last_unstale":"2024-09-13T13:51:45.160719+0000","last_undegraded":"2024-09-13T13:51:45.160719+0000","last_fullsized":"2024-09-13T13:51:45.160719+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'120","last_scrub_stamp":"2024-09-13T13:51:33.575276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:33.575276+0000","objects_scrubbed":6,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.266985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00037417499999999997,"stat_sum":{"num_bytes":15733827,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":142,"num_read_kb":112274,"num_write":65,"num_write_kb":17067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"4.5","version":"120'214","reported_seq":578,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.745458+0000","last_change":"2024-09-13T13:51:39.743055+0000","last_active":"2024-09-13T13:51:45.745458+0000","last_peered":"2024-09-13T13:51:45.745458+0000","last_clean":"2024-09-13T13:51:45.745458+0000","last_became_active":"2024-09-13T13:51:05.828626+0000","last_became_peered":"2024-09-13T13:51:05.828626+0000","last_unstale":"2024-09-13T13:51:45.745458+0000","last_undegraded":"2024-09-13T13:51:45.745458+0000","last_fullsized":"2024-09-13T13:51:45.745458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'163","last_scrub_stamp":"2024-09-13T13:51:14.489895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.489895+0000","objects_scrubbed":12,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.604698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003406994,"stat_sum":{"num_bytes":28313494,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":238,"num_read_kb":191538,"num_write":110,"num_write_kb":33417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":40416780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":315,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.745282+0000","last_change":"2024-09-13T13:51:09.467323+0000","last_active":"2024-09-13T13:51:45.745282+0000","last_peered":"2024-09-13T13:51:45.745282+0000","last_clean":"2024-09-13T13:51:45.745282+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:51:45.745282+0000","last_undegraded":"2024-09-13T13:51:45.745282+0000","last_fullsized":"2024-09-13T13:51:45.745282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:51:09.467249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.467249+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:36.764015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":224,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006606+0000","last_change":"2024-09-13T13:51:13.789246+0000","last_active":"2024-09-13T13:51:48.006606+0000","last_peered":"2024-09-13T13:51:48.006606+0000","last_clean":"2024-09-13T13:51:48.006606+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:51:48.006606+0000","last_undegraded":"2024-09-13T13:51:48.006606+0000","last_fullsized":"2024-09-13T13:51:48.006606+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:51:13.789157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.789157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:16.372966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"127'141","reported_seq":490,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006579+0000","last_change":"2024-09-13T13:51:23.855465+0000","last_active":"2024-09-13T13:51:48.006579+0000","last_peered":"2024-09-13T13:51:48.006579+0000","last_clean":"2024-09-13T13:51:48.006579+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:51:48.006579+0000","last_undegraded":"2024-09-13T13:51:48.006579+0000","last_fullsized":"2024-09-13T13:51:48.006579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'131","last_scrub_stamp":"2024-09-13T13:51:23.855357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.855357+0000","objects_scrubbed":22,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:35.255469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27764736,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":110,"num_read_kb":7463,"num_write":150,"num_write_kb":60570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"127'120","reported_seq":627,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.924955+0000","last_change":"2024-09-13T13:51:46.924955+0000","last_active":"2024-09-13T13:51:46.924955+0000","last_peered":"2024-09-13T13:51:46.924955+0000","last_clean":"2024-09-13T13:51:46.924955+0000","last_became_active":"2024-09-13T13:51:00.088253+0000","last_became_peered":"2024-09-13T13:51:00.088253+0000","last_unstale":"2024-09-13T13:51:46.924955+0000","last_undegraded":"2024-09-13T13:51:46.924955+0000","last_fullsized":"2024-09-13T13:51:46.924955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'106","last_scrub_stamp":"2024-09-13T13:51:25.488671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.488671+0000","objects_scrubbed":6,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:50.504410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.006353221,"stat_sum":{"num_bytes":11342733,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":200,"num_read_kb":171943,"num_write":65,"num_write_kb":17212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14889074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":226,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.003760+0000","last_change":"2024-09-13T13:51:26.177165+0000","last_active":"2024-09-13T13:51:48.003760+0000","last_peered":"2024-09-13T13:51:48.003760+0000","last_clean":"2024-09-13T13:51:48.003760+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:51:48.003760+0000","last_undegraded":"2024-09-13T13:51:48.003760+0000","last_fullsized":"2024-09-13T13:51:48.003760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:26.177104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:26.177104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:43.262086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"122'144","reported_seq":385,"reported_epoch":126,"state":"active+recovering+degraded","last_fresh":"2024-09-13T13:51:45.994260+0000","last_change":"2024-09-13T13:51:45.957598+0000","last_active":"2024-09-13T13:51:45.994260+0000","last_peered":"2024-09-13T13:51:45.994260+0000","last_clean":"2024-09-13T13:51:43.050674+0000","last_became_active":"2024-09-13T13:51:45.952794+0000","last_became_peered":"2024-09-13T13:51:45.952794+0000","last_unstale":"2024-09-13T13:51:45.994260+0000","last_undegraded":"2024-09-13T13:51:45.746997+0000","last_fullsized":"2024-09-13T13:51:45.994260+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:50:04.058166+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30181163,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":116,"num_read_kb":8403,"num_write":157,"num_write_kb":59191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":6401639,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":["13"],"object_location_counts":[{"shards":"13","objects":23},{"shards":"13,14","objects":5}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"127'171","reported_seq":641,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.905719+0000","last_change":"2024-09-13T13:51:46.896036+0000","last_active":"2024-09-13T13:51:46.905719+0000","last_peered":"2024-09-13T13:51:46.905719+0000","last_clean":"2024-09-13T13:51:46.905719+0000","last_became_active":"2024-09-13T13:50:29.051980+0000","last_became_peered":"2024-09-13T13:50:29.051980+0000","last_unstale":"2024-09-13T13:51:46.905719+0000","last_undegraded":"2024-09-13T13:51:46.905719+0000","last_fullsized":"2024-09-13T13:51:46.905719+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'134","last_scrub_stamp":"2024-09-13T13:51:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.183462+0000","objects_scrubbed":9,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:29.315774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039293050000000001,"stat_sum":{"num_bytes":22881648,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":173,"num_read_kb":126696,"num_write":94,"num_write_kb":24374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9395884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":223,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006589+0000","last_change":"2024-09-13T13:51:11.801324+0000","last_active":"2024-09-13T13:51:48.006589+0000","last_peered":"2024-09-13T13:51:48.006589+0000","last_clean":"2024-09-13T13:51:48.006589+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:51:48.006589+0000","last_undegraded":"2024-09-13T13:51:48.006589+0000","last_fullsized":"2024-09-13T13:51:48.006589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:11.801231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:11.801231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.494865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"117'124","reported_seq":218,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.160633+0000","last_change":"2024-09-13T13:51:25.709840+0000","last_active":"2024-09-13T13:51:45.160633+0000","last_peered":"2024-09-13T13:51:45.160633+0000","last_clean":"2024-09-13T13:51:45.160633+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:51:45.160633+0000","last_undegraded":"2024-09-13T13:51:45.160633+0000","last_fullsized":"2024-09-13T13:51:45.160633+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'121","last_scrub_stamp":"2024-09-13T13:51:25.709738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.709738+0000","objects_scrubbed":20,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:33.799047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21930253,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":97,"num_read_kb":2325,"num_write":131,"num_write_kb":48898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"120'140","reported_seq":262,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.745433+0000","last_change":"2024-09-13T13:51:39.742016+0000","last_active":"2024-09-13T13:51:45.745433+0000","last_peered":"2024-09-13T13:51:45.745433+0000","last_clean":"2024-09-13T13:51:45.745433+0000","last_became_active":"2024-09-13T13:51:02.819431+0000","last_became_peered":"2024-09-13T13:51:02.819431+0000","last_unstale":"2024-09-13T13:51:45.745433+0000","last_undegraded":"2024-09-13T13:51:45.745433+0000","last_fullsized":"2024-09-13T13:51:45.745433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'136","last_scrub_stamp":"2024-09-13T13:51:34.295925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:34.295925+0000","objects_scrubbed":9,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.498048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002288572,"stat_sum":{"num_bytes":17119931,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":187,"num_read_kb":175597,"num_write":79,"num_write_kb":23173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27682344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":238,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.160962+0000","last_change":"2024-09-13T13:51:09.226997+0000","last_active":"2024-09-13T13:51:45.160962+0000","last_peered":"2024-09-13T13:51:45.160962+0000","last_clean":"2024-09-13T13:51:45.160962+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:51:45.160962+0000","last_undegraded":"2024-09-13T13:51:45.160962+0000","last_fullsized":"2024-09-13T13:51:45.160962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:51:09.226898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.226898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:21.066559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"121'130","reported_seq":349,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:44.740384+0000","last_change":"2024-09-13T13:51:23.029600+0000","last_active":"2024-09-13T13:51:44.740384+0000","last_peered":"2024-09-13T13:51:44.740384+0000","last_clean":"2024-09-13T13:51:44.740384+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:51:44.740384+0000","last_undegraded":"2024-09-13T13:51:44.740384+0000","last_fullsized":"2024-09-13T13:51:44.740384+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'125","last_scrub_stamp":"2024-09-13T13:51:23.029522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.029522+0000","objects_scrubbed":20,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:40.590274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23551210,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":6351,"num_write":145,"num_write_kb":49287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"120'189","reported_seq":491,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.745455+0000","last_change":"2024-09-13T13:51:39.744439+0000","last_active":"2024-09-13T13:51:45.745455+0000","last_peered":"2024-09-13T13:51:45.745455+0000","last_clean":"2024-09-13T13:51:45.745455+0000","last_became_active":"2024-09-13T13:51:06.832059+0000","last_became_peered":"2024-09-13T13:51:06.832059+0000","last_unstale":"2024-09-13T13:51:45.745455+0000","last_undegraded":"2024-09-13T13:51:45.745455+0000","last_fullsized":"2024-09-13T13:51:45.745455+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T13:51:10.824926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:10.824926+0000","objects_scrubbed":10,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:32.438865+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0.0047365599999999999,"stat_sum":{"num_bytes":20000424,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":283,"num_read_kb":257944,"num_write":99,"num_write_kb":27539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":33977752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":222,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.892041+0000","last_change":"2024-09-13T13:51:13.190624+0000","last_active":"2024-09-13T13:51:46.892041+0000","last_peered":"2024-09-13T13:51:46.892041+0000","last_clean":"2024-09-13T13:51:46.892041+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:51:46.892041+0000","last_undegraded":"2024-09-13T13:51:46.892041+0000","last_fullsized":"2024-09-13T13:51:46.892041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:13.190561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.190561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.641712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"124'34","reported_seq":443,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.742803+0000","last_change":"2024-09-13T13:51:39.748214+0000","last_active":"2024-09-13T13:51:45.742803+0000","last_peered":"2024-09-13T13:51:45.742803+0000","last_clean":"2024-09-13T13:51:45.742803+0000","last_became_active":"2024-09-13T13:50:06.075820+0000","last_became_peered":"2024-09-13T13:50:06.075820+0000","last_unstale":"2024-09-13T13:51:45.742803+0000","last_undegraded":"2024-09-13T13:51:45.742803+0000","last_fullsized":"2024-09-13T13:51:45.742803+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'23","last_scrub_stamp":"2024-09-13T13:51:14.735588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.735588+0000","objects_scrubbed":1,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.339268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026792109999999999,"stat_sum":{"num_bytes":6061176,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":68109,"num_write":24,"num_write_kb":6999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":224,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.918526+0000","last_change":"2024-09-13T13:51:01.508987+0000","last_active":"2024-09-13T13:51:46.918526+0000","last_peered":"2024-09-13T13:51:46.918526+0000","last_clean":"2024-09-13T13:51:46.918526+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:51:46.918526+0000","last_undegraded":"2024-09-13T13:51:46.918526+0000","last_fullsized":"2024-09-13T13:51:46.918526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:01.508923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:01.508923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:10.969122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"120'67","reported_seq":502,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.778024+0000","last_change":"2024-09-13T13:51:45.777955+0000","last_active":"2024-09-13T13:51:45.778024+0000","last_peered":"2024-09-13T13:51:45.778024+0000","last_clean":"2024-09-13T13:51:45.778024+0000","last_became_active":"2024-09-13T13:51:45.749549+0000","last_became_peered":"2024-09-13T13:51:45.749549+0000","last_unstale":"2024-09-13T13:51:45.778024+0000","last_undegraded":"2024-09-13T13:51:45.778024+0000","last_fullsized":"2024-09-13T13:51:45.778024+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'59","last_scrub_stamp":"2024-09-13T13:51:22.810355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.810355+0000","objects_scrubbed":3,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:29.298299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003474881,"stat_sum":{"num_bytes":4720896,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":80946,"num_write":30,"num_write_kb":6524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3724338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":218,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.160941+0000","last_change":"2024-09-13T13:50:57.984106+0000","last_active":"2024-09-13T13:51:45.160941+0000","last_peered":"2024-09-13T13:51:45.160941+0000","last_clean":"2024-09-13T13:51:45.160941+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:51:45.160941+0000","last_undegraded":"2024-09-13T13:51:45.160941+0000","last_fullsized":"2024-09-13T13:51:45.160941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:57.983997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:50:57.983997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:05.913635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"124'297","reported_seq":910,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T13:51:45.742692+0000","last_change":"2024-09-13T13:51:39.757366+0000","last_active":"2024-09-13T13:51:45.742692+0000","last_peered":"2024-09-13T13:51:45.742692+0000","last_clean":"2024-09-13T13:51:45.742692+0000","last_became_active":"2024-09-13T13:50:54.737668+0000","last_became_peered":"2024-09-13T13:50:54.737668+0000","last_unstale":"2024-09-13T13:51:45.742692+0000","last_undegraded":"2024-09-13T13:51:45.742692+0000","last_fullsized":"2024-09-13T13:51:45.742692+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'211","last_scrub_stamp":"2024-09-13T13:51:13.786427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.786427+0000","objects_scrubbed":13,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:42.450996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011942006,"stat_sum":{"num_bytes":38968033,"num_objects":19,"num_object_clones":13,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":501,"num_read_kb":425408,"num_write":142,"num_write_kb":38829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":29261751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.003840+0000","last_change":"2024-09-13T13:51:22.235733+0000","last_active":"2024-09-13T13:51:48.003840+0000","last_peered":"2024-09-13T13:51:48.003840+0000","last_clean":"2024-09-13T13:51:48.003840+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:51:48.003840+0000","last_undegraded":"2024-09-13T13:51:48.003840+0000","last_fullsized":"2024-09-13T13:51:48.003840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:22.235634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.235634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:51.179878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":294208434,"num_objects":156,"num_object_clones":106,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":156,"num_whiteouts":4,"num_read":3542,"num_read_kb":2991962,"num_write":1319,"num_write_kb":356873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":219467852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442253312,"data_stored":616564027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2588,"ondisk_log_size":2588,"up":32,"acting":32,"num_store_stats":15},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":103427362,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":439,"num_read_kb":24542,"num_write":583,"num_write_kb":217946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":97731395,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":255668224,"data_stored":255233449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36208,"kb_used_data":9184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335632,"statfs":{"total":96636764160,"available":96599687168,"internally_reserved":0,"allocated":9404416,"data_stored":13198980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,6,7,8,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114468,"kb_used_data":83284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94257372,"statfs":{"total":96636764160,"available":96519548928,"internally_reserved":0,"allocated":85282816,"data_stored":92340048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":14492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":14839808,"data_stored":18224986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280538,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109604,"kb_used_data":78420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94262236,"statfs":{"total":96636764160,"available":96524529664,"internally_reserved":0,"allocated":80302080,"data_stored":108427374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84124,"kb_used_data":57100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94287716,"statfs":{"total":96636764160,"available":96550621184,"internally_reserved":0,"allocated":58470400,"data_stored":61749712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46248,"kb_used_data":19224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325592,"statfs":{"total":96636764160,"available":96589406208,"internally_reserved":0,"allocated":19685376,"data_stored":28407617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280538,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131092,"kb_used_data":99908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94240748,"statfs":{"total":96636764160,"available":96502525952,"internally_reserved":0,"allocated":102305792,"data_stored":136781264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1273856,"data_stored":854366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70912,"kb_used_data":43888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94300928,"statfs":{"total":96636764160,"available":96564150272,"internally_reserved":0,"allocated":44941312,"data_stored":48686269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46640,"kb_used_data":19616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325200,"statfs":{"total":96636764160,"available":96589004800,"internally_reserved":0,"allocated":20086784,"data_stored":27509701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313243,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78140,"kb_used_data":46956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94293700,"statfs":{"total":96636764160,"available":96556748800,"internally_reserved":0,"allocated":48082944,"data_stored":57183419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1241088,"data_stored":853093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47056,"kb_used_data":15872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324784,"statfs":{"total":96636764160,"available":96588578816,"internally_reserved":0,"allocated":16252928,"data_stored":29421592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":29772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":30486528,"data_stored":33395309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,5,6,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129396,"kb_used_data":98212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94242444,"statfs":{"total":96636764160,"available":96504262656,"internally_reserved":0,"allocated":100569088,"data_stored":125040879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46616576,"data_stored":46528781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30228480,"data_stored":30181163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":21970944,"data_stored":21930253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23760896,"data_stored":23722218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":23588864,"data_stored":23551210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":30228480,"data_stored":30181163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":59052032,"data_stored":58952491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":18655977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14913536,"data_stored":28547324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79171584,"data_stored":107561066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10575872,"data_stored":14362917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16887808,"data_stored":25492208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":18345984,"data_stored":27535430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":104173568,"data_stored":141513618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7221248,"data_stored":10594007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":101915461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":20074496,"data_stored":24277782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13533184,"data_stored":17354526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":18817024,"data_stored":26652150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":16732160,"data_stored":26136619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":24862720,"data_stored":32504118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7532544,"data_stored":13460824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:50.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:50.909 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:50.908+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:50.953 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:50.953 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:50.953 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:50.953 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:50.961 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:51.009 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:51.008+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:51.023 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:51:51.026 INFO:tasks.rados.rados.0.smithi159.stdout:1648: done (4 left) 2024-09-13T13:51:51.027 INFO:tasks.rados.rados.0.smithi159.stdout:1649: read oid 30 snap -1 2024-09-13T13:51:51.027 INFO:tasks.rados.rados.0.smithi159.stdout:1649: expect (ObjNum 301 snap 43 seq_num 301) 2024-09-13T13:51:51.027 INFO:tasks.rados.rados.0.smithi159.stdout:1650: snap_create 2024-09-13T13:51:51.027 INFO:tasks.rados.rados.0.smithi159.stdout:1643: expect (ObjNum 296 snap 41 seq_num 296) 2024-09-13T13:51:51.069 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:51.069 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:51.070 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:51.070 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:51.070 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:51.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:51.088 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:51.110 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:51.110+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:51.126 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:51:51.126 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:51:51.138 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-13T13:51:51.023994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404003878,"num_objects":277,"num_object_clones":111,"num_object_copies":554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":7,"num_read":4194,"num_read_kb":3130485,"num_write":2093,"num_write_kb":584718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":341569177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":79,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3367,"ondisk_log_size":3367,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":1071796,"kb_used_data":637316,"kb_used_omap":23,"kb_used_meta":434216,"kb_avail":1414505804,"statfs":{"total":1449551462400,"available":1448453943296,"internally_reserved":0,"allocated":652611584,"data_stored":806398368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23924,"internal_metadata":444637836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13611293,"num_objects":22,"num_object_clones":7,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":183,"num_read_kb":176144,"num_write":69,"num_write_kb":18005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":23779524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.639373"},"pg_stats":[{"pgid":"4.f","version":"129'216","reported_seq":698,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.748537+0000","last_change":"2024-09-13T13:51:48.748537+0000","last_active":"2024-09-13T13:51:48.748537+0000","last_peered":"2024-09-13T13:51:48.748537+0000","last_clean":"2024-09-13T13:51:48.748537+0000","last_became_active":"2024-09-13T13:50:06.073131+0000","last_became_peered":"2024-09-13T13:50:06.073131+0000","last_unstale":"2024-09-13T13:51:48.748537+0000","last_undegraded":"2024-09-13T13:51:48.748537+0000","last_fullsized":"2024-09-13T13:51:48.748537+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'188","last_scrub_stamp":"2024-09-13T13:51:25.039858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.039858+0000","objects_scrubbed":8,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:33.921201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00041742,"stat_sum":{"num_bytes":13877479,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":278,"num_read_kb":224319,"num_write":93,"num_write_kb":22889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.e","version":"129'253","reported_seq":819,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.767810+0000","last_change":"2024-09-13T13:51:48.767992+0000","last_active":"2024-09-13T13:51:48.767810+0000","last_peered":"2024-09-13T13:51:48.767810+0000","last_clean":"2024-09-13T13:51:48.767810+0000","last_became_active":"2024-09-13T13:51:45.749931+0000","last_became_peered":"2024-09-13T13:51:45.749931+0000","last_unstale":"2024-09-13T13:51:48.767810+0000","last_undegraded":"2024-09-13T13:51:48.767810+0000","last_fullsized":"2024-09-13T13:51:48.767810+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'178","last_scrub_stamp":"2024-09-13T13:51:13.109829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.109829+0000","objects_scrubbed":9,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:32.194014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0025224980000000002,"stat_sum":{"num_bytes":21215512,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":394,"num_read_kb":308562,"num_write":116,"num_write_kb":29909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":14756273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.d","version":"129'57","reported_seq":488,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.065668+0000","last_change":"2024-09-13T13:51:49.065825+0000","last_active":"2024-09-13T13:51:49.065668+0000","last_peered":"2024-09-13T13:51:49.065668+0000","last_clean":"2024-09-13T13:51:49.065668+0000","last_became_active":"2024-09-13T13:50:06.075524+0000","last_became_peered":"2024-09-13T13:50:06.075524+0000","last_unstale":"2024-09-13T13:51:49.065668+0000","last_undegraded":"2024-09-13T13:51:49.065668+0000","last_fullsized":"2024-09-13T13:51:49.065668+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'38","last_scrub_stamp":"2024-09-13T13:51:21.332048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:21.332048+0000","objects_scrubbed":3,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:22.242560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019528480000000001,"stat_sum":{"num_bytes":6671280,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":71,"num_read_kb":60736,"num_write":25,"num_write_kb":6222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.c","version":"129'229","reported_seq":771,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.911923+0000","last_change":"2024-09-13T13:51:49.423185+0000","last_active":"2024-09-13T13:51:49.911923+0000","last_peered":"2024-09-13T13:51:49.911923+0000","last_clean":"2024-09-13T13:51:49.911923+0000","last_became_active":"2024-09-13T13:50:28.729220+0000","last_became_peered":"2024-09-13T13:50:28.729220+0000","last_unstale":"2024-09-13T13:51:49.911923+0000","last_undegraded":"2024-09-13T13:51:49.911923+0000","last_fullsized":"2024-09-13T13:51:49.911923+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'162","last_scrub_stamp":"2024-09-13T13:51:12.406192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:12.406192+0000","objects_scrubbed":8,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:32.366956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.35932499200000001,"stat_sum":{"num_bytes":22075196,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":326,"num_read_kb":290406,"num_write":107,"num_write_kb":26915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11131276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.b","version":"127'269","reported_seq":719,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006828+0000","last_change":"2024-09-13T13:51:48.007022+0000","last_active":"2024-09-13T13:51:48.006828+0000","last_peered":"2024-09-13T13:51:48.006828+0000","last_clean":"2024-09-13T13:51:48.006828+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:51:48.006828+0000","last_undegraded":"2024-09-13T13:51:48.006828+0000","last_fullsized":"2024-09-13T13:51:48.006828+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'224","last_scrub_stamp":"2024-09-13T13:51:25.879055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.879055+0000","objects_scrubbed":16,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:52.165940+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00036324399999999998,"stat_sum":{"num_bytes":34258690,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":223,"num_read_kb":154035,"num_write":158,"num_write_kb":43790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.a","version":"128'201","reported_seq":779,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.005887+0000","last_change":"2024-09-13T13:51:48.006066+0000","last_active":"2024-09-13T13:51:48.005887+0000","last_peered":"2024-09-13T13:51:48.005887+0000","last_clean":"2024-09-13T13:51:48.005887+0000","last_became_active":"2024-09-13T13:51:02.821659+0000","last_became_peered":"2024-09-13T13:51:02.821659+0000","last_unstale":"2024-09-13T13:51:48.005887+0000","last_undegraded":"2024-09-13T13:51:48.005887+0000","last_fullsized":"2024-09-13T13:51:48.005887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'158","last_scrub_stamp":"2024-09-13T13:51:24.201568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:24.201568+0000","objects_scrubbed":11,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:49.479666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019803059999999998,"stat_sum":{"num_bytes":26967303,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":337,"num_read_kb":330466,"num_write":93,"num_write_kb":27234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":34232380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.9","version":"127'42","reported_seq":434,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:47.707070+0000","last_change":"2024-09-13T13:51:46.749938+0000","last_active":"2024-09-13T13:51:47.707070+0000","last_peered":"2024-09-13T13:51:47.707070+0000","last_clean":"2024-09-13T13:51:47.707070+0000","last_became_active":"2024-09-13T13:50:12.197810+0000","last_became_peered":"2024-09-13T13:50:12.197810+0000","last_unstale":"2024-09-13T13:51:47.707070+0000","last_undegraded":"2024-09-13T13:51:47.707070+0000","last_fullsized":"2024-09-13T13:51:47.707070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'21","last_scrub_stamp":"2024-09-13T13:51:21.631083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:21.631083+0000","objects_scrubbed":1,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:28.265320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00297263,"stat_sum":{"num_bytes":3783473,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":51272,"num_write":22,"num_write_kb":4780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5],"acting":[15,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"4.8","version":"129'145","reported_seq":582,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.752135+0000","last_change":"2024-09-13T13:51:48.752135+0000","last_active":"2024-09-13T13:51:48.752135+0000","last_peered":"2024-09-13T13:51:48.752135+0000","last_clean":"2024-09-13T13:51:48.752135+0000","last_became_active":"2024-09-13T13:50:06.075916+0000","last_became_peered":"2024-09-13T13:50:06.075916+0000","last_unstale":"2024-09-13T13:51:48.752135+0000","last_undegraded":"2024-09-13T13:51:48.752135+0000","last_fullsized":"2024-09-13T13:51:48.752135+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'120","last_scrub_stamp":"2024-09-13T13:51:33.575276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:33.575276+0000","objects_scrubbed":6,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:39.113820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021584799999999999,"stat_sum":{"num_bytes":13669085,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":143,"num_read_kb":112274,"num_write":67,"num_write_kb":17067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"4.5","version":"129'229","reported_seq":622,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.911856+0000","last_change":"2024-09-13T13:51:48.751691+0000","last_active":"2024-09-13T13:51:49.911856+0000","last_peered":"2024-09-13T13:51:49.911856+0000","last_clean":"2024-09-13T13:51:49.911856+0000","last_became_active":"2024-09-13T13:51:05.828626+0000","last_became_peered":"2024-09-13T13:51:05.828626+0000","last_unstale":"2024-09-13T13:51:49.911856+0000","last_undegraded":"2024-09-13T13:51:49.911856+0000","last_fullsized":"2024-09-13T13:51:49.911856+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'163","last_scrub_stamp":"2024-09-13T13:51:14.489895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.489895+0000","objects_scrubbed":12,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:39.848102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020072010000000001,"stat_sum":{"num_bytes":30113613,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":266,"num_read_kb":219702,"num_write":114,"num_write_kb":34521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":40416780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"1.0","version":"27'112","reported_seq":321,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.749503+0000","last_change":"2024-09-13T13:51:09.467323+0000","last_active":"2024-09-13T13:51:48.749503+0000","last_peered":"2024-09-13T13:51:48.749503+0000","last_clean":"2024-09-13T13:51:48.749503+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:51:48.749503+0000","last_undegraded":"2024-09-13T13:51:48.749503+0000","last_fullsized":"2024-09-13T13:51:48.749503+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:51:09.467249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.467249+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:36.764015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":224,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006606+0000","last_change":"2024-09-13T13:51:13.789246+0000","last_active":"2024-09-13T13:51:48.006606+0000","last_peered":"2024-09-13T13:51:48.006606+0000","last_clean":"2024-09-13T13:51:48.006606+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:51:48.006606+0000","last_undegraded":"2024-09-13T13:51:48.006606+0000","last_fullsized":"2024-09-13T13:51:48.006606+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:51:13.789157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.789157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:16.372966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"127'141","reported_seq":490,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006579+0000","last_change":"2024-09-13T13:51:23.855465+0000","last_active":"2024-09-13T13:51:48.006579+0000","last_peered":"2024-09-13T13:51:48.006579+0000","last_clean":"2024-09-13T13:51:48.006579+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:51:48.006579+0000","last_undegraded":"2024-09-13T13:51:48.006579+0000","last_fullsized":"2024-09-13T13:51:48.006579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'131","last_scrub_stamp":"2024-09-13T13:51:23.855357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.855357+0000","objects_scrubbed":22,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:35.255469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27764736,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":110,"num_read_kb":7463,"num_write":150,"num_write_kb":60570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"127'120","reported_seq":627,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.924955+0000","last_change":"2024-09-13T13:51:46.924955+0000","last_active":"2024-09-13T13:51:46.924955+0000","last_peered":"2024-09-13T13:51:46.924955+0000","last_clean":"2024-09-13T13:51:46.924955+0000","last_became_active":"2024-09-13T13:51:00.088253+0000","last_became_peered":"2024-09-13T13:51:00.088253+0000","last_unstale":"2024-09-13T13:51:46.924955+0000","last_undegraded":"2024-09-13T13:51:46.924955+0000","last_fullsized":"2024-09-13T13:51:46.924955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'106","last_scrub_stamp":"2024-09-13T13:51:25.488671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.488671+0000","objects_scrubbed":6,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:50.504410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.006353221,"stat_sum":{"num_bytes":11342733,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":200,"num_read_kb":171943,"num_write":65,"num_write_kb":17212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14889074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":226,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.003760+0000","last_change":"2024-09-13T13:51:26.177165+0000","last_active":"2024-09-13T13:51:48.003760+0000","last_peered":"2024-09-13T13:51:48.003760+0000","last_clean":"2024-09-13T13:51:48.003760+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:51:48.003760+0000","last_undegraded":"2024-09-13T13:51:48.003760+0000","last_fullsized":"2024-09-13T13:51:48.003760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:26.177104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:26.177104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:43.262086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"129'152","reported_seq":481,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.843053+0000","last_change":"2024-09-13T13:51:49.843053+0000","last_active":"2024-09-13T13:51:49.843053+0000","last_peered":"2024-09-13T13:51:49.843053+0000","last_clean":"2024-09-13T13:51:49.843053+0000","last_became_active":"2024-09-13T13:51:45.952794+0000","last_became_peered":"2024-09-13T13:51:45.952794+0000","last_unstale":"2024-09-13T13:51:49.843053+0000","last_undegraded":"2024-09-13T13:51:49.843053+0000","last_fullsized":"2024-09-13T13:51:49.843053+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'152","last_scrub_stamp":"2024-09-13T13:51:49.842939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:49.842939+0000","objects_scrubbed":34,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:00.129584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32432002,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":121,"num_read_kb":8403,"num_write":168,"num_write_kb":61391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30181163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"127'171","reported_seq":641,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.905719+0000","last_change":"2024-09-13T13:51:46.896036+0000","last_active":"2024-09-13T13:51:46.905719+0000","last_peered":"2024-09-13T13:51:46.905719+0000","last_clean":"2024-09-13T13:51:46.905719+0000","last_became_active":"2024-09-13T13:50:29.051980+0000","last_became_peered":"2024-09-13T13:50:29.051980+0000","last_unstale":"2024-09-13T13:51:46.905719+0000","last_undegraded":"2024-09-13T13:51:46.905719+0000","last_fullsized":"2024-09-13T13:51:46.905719+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'134","last_scrub_stamp":"2024-09-13T13:51:23.183462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.183462+0000","objects_scrubbed":9,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:29.315774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039293050000000001,"stat_sum":{"num_bytes":22881648,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":173,"num_read_kb":126696,"num_write":94,"num_write_kb":24374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9395884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"17","length":"1"},{"start":"1b","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":223,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.006589+0000","last_change":"2024-09-13T13:51:11.801324+0000","last_active":"2024-09-13T13:51:48.006589+0000","last_peered":"2024-09-13T13:51:48.006589+0000","last_clean":"2024-09-13T13:51:48.006589+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:51:48.006589+0000","last_undegraded":"2024-09-13T13:51:48.006589+0000","last_fullsized":"2024-09-13T13:51:48.006589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:11.801231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:11.801231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:15.494865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"129'129","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.796125+0000","last_change":"2024-09-13T13:51:25.709840+0000","last_active":"2024-09-13T13:51:48.796125+0000","last_peered":"2024-09-13T13:51:48.796125+0000","last_clean":"2024-09-13T13:51:48.796125+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:51:48.796125+0000","last_undegraded":"2024-09-13T13:51:48.796125+0000","last_fullsized":"2024-09-13T13:51:48.796125+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'121","last_scrub_stamp":"2024-09-13T13:51:25.709738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:25.709738+0000","objects_scrubbed":20,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:33.799047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22967527,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":102,"num_read_kb":2325,"num_write":141,"num_write_kb":49913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"129'147","reported_seq":293,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.536975+0000","last_change":"2024-09-13T13:51:48.750091+0000","last_active":"2024-09-13T13:51:49.536975+0000","last_peered":"2024-09-13T13:51:49.536975+0000","last_clean":"2024-09-13T13:51:49.536975+0000","last_became_active":"2024-09-13T13:51:02.819431+0000","last_became_peered":"2024-09-13T13:51:02.819431+0000","last_unstale":"2024-09-13T13:51:49.536975+0000","last_undegraded":"2024-09-13T13:51:49.536975+0000","last_fullsized":"2024-09-13T13:51:49.536975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'136","last_scrub_stamp":"2024-09-13T13:51:34.295925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:34.295925+0000","objects_scrubbed":9,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:50.034889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00045085699999999998,"stat_sum":{"num_bytes":18413640,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":197,"num_read_kb":181301,"num_write":84,"num_write_kb":24438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27682344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.061988+0000","last_change":"2024-09-13T13:51:09.226997+0000","last_active":"2024-09-13T13:51:49.061988+0000","last_peered":"2024-09-13T13:51:49.061988+0000","last_clean":"2024-09-13T13:51:49.061988+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:51:49.061988+0000","last_undegraded":"2024-09-13T13:51:49.061988+0000","last_fullsized":"2024-09-13T13:51:49.061988+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:51:09.226898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.226898+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:21.066559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"129'133","reported_seq":360,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.753483+0000","last_change":"2024-09-13T13:51:23.029600+0000","last_active":"2024-09-13T13:51:48.753483+0000","last_peered":"2024-09-13T13:51:48.753483+0000","last_clean":"2024-09-13T13:51:48.753483+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:51:48.753483+0000","last_undegraded":"2024-09-13T13:51:48.753483+0000","last_fullsized":"2024-09-13T13:51:48.753483+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'125","last_scrub_stamp":"2024-09-13T13:51:23.029522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.029522+0000","objects_scrubbed":20,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:40.590274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23992915,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":118,"num_read_kb":6351,"num_write":149,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"127'193","reported_seq":518,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.750145+0000","last_change":"2024-09-13T13:51:48.750330+0000","last_active":"2024-09-13T13:51:48.750145+0000","last_peered":"2024-09-13T13:51:48.750145+0000","last_clean":"2024-09-13T13:51:48.750145+0000","last_became_active":"2024-09-13T13:51:06.832059+0000","last_became_peered":"2024-09-13T13:51:06.832059+0000","last_unstale":"2024-09-13T13:51:48.750145+0000","last_undegraded":"2024-09-13T13:51:48.750145+0000","last_fullsized":"2024-09-13T13:51:48.750145+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-09-13T13:51:10.824926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:10.824926+0000","objects_scrubbed":10,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:17.027147+0000","scrub_duration":378,"objects_trimmed":0,"snaptrim_duration":0.00057592500000000003,"stat_sum":{"num_bytes":20000424,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":292,"num_read_kb":273286,"num_write":99,"num_write_kb":27539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":33977752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":222,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.892041+0000","last_change":"2024-09-13T13:51:13.190624+0000","last_active":"2024-09-13T13:51:46.892041+0000","last_peered":"2024-09-13T13:51:46.892041+0000","last_clean":"2024-09-13T13:51:46.892041+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:51:46.892041+0000","last_undegraded":"2024-09-13T13:51:46.892041+0000","last_fullsized":"2024-09-13T13:51:46.892041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:13.190561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.190561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:41.641712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"129'38","reported_seq":468,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.770791+0000","last_change":"2024-09-13T13:51:48.770935+0000","last_active":"2024-09-13T13:51:48.770791+0000","last_peered":"2024-09-13T13:51:48.770791+0000","last_clean":"2024-09-13T13:51:48.770791+0000","last_became_active":"2024-09-13T13:50:06.075820+0000","last_became_peered":"2024-09-13T13:50:06.075820+0000","last_unstale":"2024-09-13T13:51:48.770791+0000","last_undegraded":"2024-09-13T13:51:48.770791+0000","last_fullsized":"2024-09-13T13:51:48.770791+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'23","last_scrub_stamp":"2024-09-13T13:51:14.735588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:14.735588+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:29.937209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0053497650000000002,"stat_sum":{"num_bytes":6061176,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":68109,"num_write":24,"num_write_kb":6999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":224,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.918526+0000","last_change":"2024-09-13T13:51:01.508987+0000","last_active":"2024-09-13T13:51:46.918526+0000","last_peered":"2024-09-13T13:51:46.918526+0000","last_clean":"2024-09-13T13:51:46.918526+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:51:46.918526+0000","last_undegraded":"2024-09-13T13:51:46.918526+0000","last_fullsized":"2024-09-13T13:51:46.918526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:01.508923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:01.508923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:10.969122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"127'69","reported_seq":525,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.842431+0000","last_change":"2024-09-13T13:51:48.842652+0000","last_active":"2024-09-13T13:51:48.842431+0000","last_peered":"2024-09-13T13:51:48.842431+0000","last_clean":"2024-09-13T13:51:48.842431+0000","last_became_active":"2024-09-13T13:51:45.749549+0000","last_became_peered":"2024-09-13T13:51:45.749549+0000","last_unstale":"2024-09-13T13:51:48.842431+0000","last_undegraded":"2024-09-13T13:51:48.842431+0000","last_fullsized":"2024-09-13T13:51:48.842431+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'59","last_scrub_stamp":"2024-09-13T13:51:22.810355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.810355+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:28.284922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043937799999999998,"stat_sum":{"num_bytes":4720896,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":80946,"num_write":30,"num_write_kb":6524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3724338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.062004+0000","last_change":"2024-09-13T13:50:57.984106+0000","last_active":"2024-09-13T13:51:49.062004+0000","last_peered":"2024-09-13T13:51:49.062004+0000","last_clean":"2024-09-13T13:51:49.062004+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:51:49.062004+0000","last_undegraded":"2024-09-13T13:51:49.062004+0000","last_fullsized":"2024-09-13T13:51:49.062004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:50:57.983997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:50:57.983997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:05.913635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"129'318","reported_seq":957,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T13:51:49.918621+0000","last_change":"2024-09-13T13:51:48.774712+0000","last_active":"2024-09-13T13:51:49.918621+0000","last_peered":"2024-09-13T13:51:49.918621+0000","last_clean":"2024-09-13T13:51:49.918621+0000","last_became_active":"2024-09-13T13:50:54.737668+0000","last_became_peered":"2024-09-13T13:50:54.737668+0000","last_unstale":"2024-09-13T13:51:49.918621+0000","last_undegraded":"2024-09-13T13:51:49.918621+0000","last_fullsized":"2024-09-13T13:51:49.918621+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'211","last_scrub_stamp":"2024-09-13T13:51:13.786427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.786427+0000","objects_scrubbed":13,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:20.145042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0091549460000000006,"stat_sum":{"num_bytes":40204163,"num_objects":20,"num_object_clones":14,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":526,"num_read_kb":451817,"num_write":147,"num_write_kb":40038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":29261751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"17","length":"1"},{"start":"26","length":"1"},{"start":"2b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T13:51:48.003840+0000","last_change":"2024-09-13T13:51:22.235733+0000","last_active":"2024-09-13T13:51:48.003840+0000","last_peered":"2024-09-13T13:51:48.003840+0000","last_clean":"2024-09-13T13:51:48.003840+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:51:48.003840+0000","last_undegraded":"2024-09-13T13:51:48.003840+0000","last_fullsized":"2024-09-13T13:51:48.003840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:22.235634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.235634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:51.179878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":296256311,"num_objects":161,"num_object_clones":111,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":7,"num_read":3657,"num_read_kb":3105870,"num_write":1338,"num_write_kb":360451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":219467852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":79,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":449626112,"data_stored":626904386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2697,"ondisk_log_size":2697,"up":32,"acting":32,"num_store_stats":15},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":107157180,"num_objects":112,"num_object_clones":0,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":451,"num_read_kb":24542,"num_write":608,"num_write_kb":221593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":121510919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263258112,"data_stored":262798065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":555,"ondisk_log_size":555,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36208,"kb_used_data":9184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335632,"statfs":{"total":96636764160,"available":96599687168,"internally_reserved":0,"allocated":9404416,"data_stored":13198980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,6,7,8,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247835,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114468,"kb_used_data":83284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94257372,"statfs":{"total":96636764160,"available":96519548928,"internally_reserved":0,"allocated":85282816,"data_stored":92340048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":14492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":14839808,"data_stored":18224986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280539,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111132,"kb_used_data":79948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260708,"statfs":{"total":96636764160,"available":96522964992,"internally_reserved":0,"allocated":81866752,"data_stored":111269931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87424,"kb_used_data":60400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94284416,"statfs":{"total":96636764160,"available":96547241984,"internally_reserved":0,"allocated":61849600,"data_stored":64145464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48800,"kb_used_data":21776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323040,"statfs":{"total":96636764160,"available":96586792960,"internally_reserved":0,"allocated":22298624,"data_stored":32845005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138552,"kb_used_data":107368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94233288,"statfs":{"total":96636764160,"available":96494886912,"internally_reserved":0,"allocated":109944832,"data_stored":147993973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1323008,"data_stored":887352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72500,"kb_used_data":45476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94299340,"statfs":{"total":96636764160,"available":96562524160,"internally_reserved":0,"allocated":46567424,"data_stored":50868857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":19664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":20135936,"data_stored":26453831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313244,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80548,"kb_used_data":49364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94291292,"statfs":{"total":96636764160,"available":96554283008,"internally_reserved":0,"allocated":50548736,"data_stored":59459068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28236,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343604,"statfs":{"total":96636764160,"available":96607850496,"internally_reserved":0,"allocated":1241088,"data_stored":853093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47056,"kb_used_data":15872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324784,"statfs":{"total":96636764160,"available":96588578816,"internally_reserved":0,"allocated":16252928,"data_stored":29421592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":29772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":30486528,"data_stored":33395309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,5,6,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129396,"kb_used_data":98212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94242444,"statfs":{"total":96636764160,"available":96504262656,"internally_reserved":0,"allocated":100569088,"data_stored":125040879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51499008,"data_stored":51400853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30228480,"data_stored":30181163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":21970944,"data_stored":21930253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":23760896,"data_stored":23722218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24035328,"data_stored":23992915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":32489472,"data_stored":32432002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":59052032,"data_stored":58952491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":18655977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14913536,"data_stored":28547324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80535552,"data_stored":110379544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9023488,"data_stored":11853611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16887808,"data_stored":25492208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20922368,"data_stored":31948008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":108023808,"data_stored":146512477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7221248,"data_stored":10594007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":75517952,"data_stored":101915461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":21209088,"data_stored":25986388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13533184,"data_stored":17354526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":18817024,"data_stored":25563294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":16732160,"data_stored":26136619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":24862720,"data_stored":32504118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7532544,"data_stored":13460824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:51.140 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:51:51.141 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-09-13T13:51:51.157 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 551542 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:51.184 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 122.640113423, 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:51:51.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:51:51.189 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 120.98067135700001, 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:51:51.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:51:51.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.194 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 120.979283133, 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:51:51.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:51:51.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:51:51.199 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 120.57885499, 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:51.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:51:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:51:51.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 98.265297833999995, 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.206 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:51:51.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:51:51.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 97.411459085999994, 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.212 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:51:51.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:51:51.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:51:51.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:51:51.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:51:51.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.220 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 92.992024216999994, 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.221 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:51:51.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.226 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 92.991976266999998, 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.227 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:51.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.229 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:51:51.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:51:51.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:51:51.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:51:51.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.234 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 91.303513339999995, 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.235 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:51.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:51:51.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:51:51.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:51:51.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:51:51.244 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.607906213000007, 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:51:51.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:51:51.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:51:51.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:51:51.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:51:51.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:51:51.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:51:51.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:51:51.252 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.880203938999998, 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.253 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:51:51.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:51:51.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:51.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.854694533999997, 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.260 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:51:51.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.854028796999998, 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.265 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:51:51.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 63.853987781000001, 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:51:51.271 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:51.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:51:51.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.034850873000003, 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:51:51.277 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:51.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:51:51.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:51:51.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3275 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.239871780000001, 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061117630000000001, 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.285 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3275 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942258+0000", 2024-09-13T13:51:51.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9369999999999998e-06 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942265+0000", 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-06 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942282+0000", 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6880000000000001e-05 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942292+0000", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8689999999999999e-06 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7336000000000005e-05 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942388+0000", 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3090000000000001e-06 2024-09-13T13:51:51.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945082+0000", 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026939160000000002 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945163+0000", 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0728000000000004e-05 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945384+0000", 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221375 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948177+0000", 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027923119999999999 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948231+0000", 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:51.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948299+0000", 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8300000000000007e-05 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948368+0000", 2024-09-13T13:51:51.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8872000000000001e-05 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3276 4.9 4:977d569f:::smithi15938319-22:head [rollback 12] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.239807294999999, 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082025069999999995, 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.293 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3276 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942325+0000", 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.512e-06 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942335+0000", 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0016e-05 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942341+0000", 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942471+0000", 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013015300000000001 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942478+0000", 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.55e-06 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946541+0000", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040631779999999998 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946640+0000", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8694000000000001e-05 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946923+0000", 2024-09-13T13:51:51.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028351899999999998 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948388+0000", 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001464414 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950419+0000", 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020313689999999999 2024-09-13T13:51:51.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950465+0000", 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5519999999999998e-05 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950523+0000", 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.842e-05 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.300 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3277 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.239748974000001, 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010801006, 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.301 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3277 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:51.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942380+0000", 2024-09-13T13:51:51.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942384+0000", 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1949999999999996e-06 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942391+0000", 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0339999999999999e-06 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942396+0000", 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8409999999999999e-06 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942508+0000", 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112458 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942512+0000", 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.579e-06 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947640+0000", 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051287859999999998 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947737+0000", 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6329999999999994e-05 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947934+0000", 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019687900000000001 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948478+0000", 2024-09-13T13:51:51.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000544448 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953066+0000", 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00458831 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953129+0000", 2024-09-13T13:51:51.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2280999999999995e-05 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953180+0000", 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3520 4.9 4:977d569f:::smithi15938319-22:head [read 627200~1132544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-13T13:51:51.309 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.207545843, 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062848280000000001, 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3520 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.310 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974583+0000", 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974589+0000", 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4879999999999998e-06 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974600+0000", 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1232e-05 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974609+0000", 2024-09-13T13:51:51.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3430000000000002e-06 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974667+0000", 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7388e-05 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974672+0000", 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6930000000000004e-06 2024-09-13T13:51:51.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979695+0000", 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050231449999999997 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979758+0000", 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2232999999999999e-05 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.980867+0000", 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001109229 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3660 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 2c=[2c,2b,2a,29,28,27,26,25,24,23,22,1e,1a,14,12] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.4788035370000001, 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068015810000000001, 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:51.316 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3660 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703325+0000", 2024-09-13T13:51:51.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.91e-07 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703326+0000", 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6759999999999999e-06 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703331+0000", 2024-09-13T13:51:51.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6879999999999998e-06 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703335+0000", 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8940000000000003e-06 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706175+0000", 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028405430000000001 2024-09-13T13:51:51.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706229+0000", 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3813000000000001e-05 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:51.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706343+0000", 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114123 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706835+0000", 2024-09-13T13:51:51.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049211600000000002 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710024+0000", 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003188116 2024-09-13T13:51:51.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710080+0000", 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6168999999999997e-05 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:51.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710126+0000", 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6152000000000001e-05 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:51.327 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:51.330 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:51.330 INFO:tasks.rados.rados.0.smithi159.stdout:1645: expect (ObjNum 199 snap 18 seq_num 199) 2024-09-13T13:51:51.331 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 657884 2024-09-13T13:51:51.331 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:51.328+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:51.342 INFO:tasks.rados.rados.0.smithi159.stdout:1647: finishing set_chunk to oid 32 2024-09-13T13:51:51.342 INFO:tasks.rados.rados.0.smithi159.stdout:1647: got expected ERANGE (src dne) 2024-09-13T13:51:51.342 INFO:tasks.rados.rados.0.smithi159.stdout:1644: expect (ObjNum 210 snap 22 seq_num 210) 2024-09-13T13:51:51.430 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:51.428+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:51.438 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:51.438 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:51.438 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:51.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:51.497 INFO:tasks.rados.rados.0.smithi159.stdout:1649: expect (ObjNum 301 snap 43 seq_num 301) 2024-09-13T13:51:51.524 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 28018 2024-09-13T13:51:51.530 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:51.529+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:51.554 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:51.554 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:51.554 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:51.554 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:51.554 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:51.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:51.631 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:51.630+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:51.670 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:51.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 87.315635352000001, 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:51:51.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 86.676183171000005, 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:51.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:51:51.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 86.673586213999997, 2024-09-13T13:51:51.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:51:51.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.933876471000005, 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.684 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:51:51.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.920431746000006, 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:51.688 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.691 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 70.025485032999995, 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:51:51.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:51:51.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.981769948999997, 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.696 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.787956366000003, 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:51:51.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 59.907564522999998, 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.705 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:51:51.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 56.692473178, 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:51:51.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 56.691594748999997, 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:51:51.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:51:51.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.718 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 56.667879251000002, 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:51.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:51:51.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.723 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 54.894517202000003, 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:51:51.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.728 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:51:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 54.893916312999998, 2024-09-13T13:51:51.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:51:51.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 54.886763735999999, 2024-09-13T13:51:51.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:51:51.740 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 54.580999321999997, 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:51.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 54.573367789999999, 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:51:51.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:51.750 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.861896647000002, 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:51:51.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.860961680000003, 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:51:51.755 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:51:51.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.759 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.850278846000002, 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:51.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:51:51.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:51:51.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:51.764 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:51.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:51.767 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:51.764+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:51.867 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:51.866+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:51.874 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:51.874 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:51.874 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:51.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:51.968 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:51.967+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:51.989 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:51.990 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:51.990 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:51.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:52.028 INFO:tasks.rados.rados.0.smithi159.stdout:1643: done (5 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1644: done (4 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1645: done (3 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1647: done (2 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1649: done (1 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1650: done (0 left) 2024-09-13T13:51:52.029 INFO:tasks.rados.rados.0.smithi159.stdout:1651: delete oid 43 current snap is 44 2024-09-13T13:51:52.038 INFO:tasks.rados.rados.0.smithi159.stdout:1651: done (0 left) 2024-09-13T13:51:52.038 INFO:tasks.rados.rados.0.smithi159.stdout:1652: read oid 17 snap -1 2024-09-13T13:51:52.038 INFO:tasks.rados.rados.0.smithi159.stdout:1652: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:51:52.038 INFO:tasks.rados.rados.0.smithi159.stdout:1653: read oid 20 snap -1 2024-09-13T13:51:52.038 INFO:tasks.rados.rados.0.smithi159.stdout:1653: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:51:52.039 INFO:tasks.rados.rados.0.smithi159.stdout:1654: tier_flush oid 11 2024-09-13T13:51:52.039 INFO:tasks.rados.rados.0.smithi159.stdout:1654: tier_flush oid 11 snap -1 2024-09-13T13:51:52.039 INFO:tasks.rados.rados.0.smithi159.stdout:1655: set_chunk oid 1 target oid 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:1655: set_chunk oid 1 offset: 952832 length: 1588736 target oid f08e1b8e8b3453ce0083bbca44c37597553d1dcf8a12562eca59b00d04e71f2f offset: 0 snap -1 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:1656: tier_flush oid 28 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:1656: tier_flush oid 28 snap -1 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:1654: got (16) Device or resource busy 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 11 v 192 (ObjNum 272 snap 36 seq_num 272) unflushed 37 2024-09-13T13:51:52.061 INFO:tasks.rados.rados.0.smithi159.stdout:1654: done (4 left) 2024-09-13T13:51:52.062 INFO:tasks.rados.rados.0.smithi159.stdout:1657: write (excl) oid 26 current snap is 44 2024-09-13T13:51:52.062 INFO:tasks.rados.rados.0.smithi159.stdout:1657: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-13T13:51:52.068 INFO:tasks.rados.rados.0.smithi159.stdout:1657: writing smithi15938319-26 from 793363 to 1499424 tid 1 2024-09-13T13:51:52.069 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:52.068+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1657: writing smithi15938319-26 from 2282230 to 3078325 tid 2 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1658: read oid 50 snap -1 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1658: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1659: tier_flush oid 32 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1659: tier_flush oid 32 snap -1 2024-09-13T13:51:52.077 INFO:tasks.rados.rados.0.smithi159.stdout:1653: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:51:52.082 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:52.080+0000 7fc16a180640 -1 osd.11 130 handle_osd_map injecting map crc failure 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.941441612999995, 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.109 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.110 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.111 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:51:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:51:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.115 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.941415620000001, 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.116 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:52.117 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.118 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:51:52.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.120 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:51:52.121 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.836106798000003, 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:52.122 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.123 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:51:52.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.125 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.540071620999996, 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.126 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.127 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:51:52.128 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.129 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:51:52.130 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.131 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.132 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.540057121000004, 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.133 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.134 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.135 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:51:52.136 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:51:52.137 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.140 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.537803681, 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:52.141 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.142 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.143 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.144 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.415380393000007, 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:51:52.145 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:52.146 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.147 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:51:52.148 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.149 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.150 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.151 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.415349574000004, 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.152 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.153 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.154 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:51:52.155 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.156 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.157 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.413191466000001, 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:52.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:51:52.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.203657268000001, 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.162 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:51:52.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:51:52.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:52.166 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 98.337679735999998, 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:52.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:51:52.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:51:52.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.172 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 98.324004021999997, 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:51:52.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.308102437999999, 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.177 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:51:52.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:51:52.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.184 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.299810942999997, 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:51:52.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.295784021999999, 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.193 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:51:52.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:51:52.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.295459065999999, 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.201 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.205 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:51:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.208 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 50.295427541000002, 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.209 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.211 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:51:52.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 47.267737539999999, 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.216 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:51:52.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3255 4.a 4:5004af91:::smithi15938319-34:head [append 0~0] snapc 23=[23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.359154683, 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.285444431, 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3255 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.220 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.221 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747900+0000", 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1220000000000002e-06 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747906+0000", 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5979999999999996e-06 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747911+0000", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4e-06 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750840+0000", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029289250000000002 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750936+0000", 2024-09-13T13:51:52.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5449e-05 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751081+0000", 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014517800000000001 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751499+0000", 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00041844799999999998 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033276+0000", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28177714300000001 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033312+0000", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5370999999999997e-05 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033342+0000", 2024-09-13T13:51:52.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0457999999999999e-05 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:3255 4.a e112/85 ondisk, result = 0)", 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 23.354515589000002, 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.280808787, 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.226 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.227 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752540+0000", 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9849999999999998e-06 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033199+0000", 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28065851600000002 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033203+0000", 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5029999999999999e-06 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033267+0000", 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3596999999999994e-05 2024-09-13T13:51:52.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033273+0000", 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.789e-06 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033346+0000", 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2960999999999995e-05 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.230 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.231 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.231 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:52.232 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:52.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:52.233 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:52.231+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:52.237 INFO:tasks.rados.rados.0.smithi159.stdout:1652: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:51:52.263 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1714879 2024-09-13T13:51:52.333 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:52.332+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:52.333 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2827039 2024-09-13T13:51:52.341 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:52.344 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:51:52.344 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:52.349 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:52.406 INFO:tasks.rados.rados.0.smithi159.stdout:1656: got (16) Device or resource busy 2024-09-13T13:51:52.406 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 28 v 149 (ObjNum 273 snap 0 seq_num 1540536720) unflushed 32 2024-09-13T13:51:52.406 INFO:tasks.rados.rados.0.smithi159.stdout:1655: finishing set_chunk to oid 1 2024-09-13T13:51:52.406 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 1 set_chunk returned error code -95 offset: 952832 length: 1588736 2024-09-13T13:51:52.406 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 0 (ObjNum 287 snap 38 seq_num 287) dirty exists 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1657: finishing write tid 1 to smithi15938319-26 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1659: got (16) Device or resource busy 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 32 v 0 (ObjNum 209 snap 0 seq_num 1063180440) unflushed 43 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1657: finishing write tid 2 to smithi15938319-26 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1652: done (6 left) 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1653: done (5 left) 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1655: done (4 left) 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1656: done (3 left) 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1659: done (2 left) 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1660: tier_promote oid 25 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1657: finishing write tid 3 to smithi15938319-26 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1657: finishing write tid 4 to smithi15938319-26 2024-09-13T13:51:52.407 INFO:tasks.rados.rados.0.smithi159.stdout:1657: oid 26 updating version 0 to 124 2024-09-13T13:51:52.408 INFO:tasks.rados.rados.0.smithi159.stdout:1657: oid 26 updating version 124 to 125 2024-09-13T13:51:52.408 INFO:tasks.rados.rados.0.smithi159.stdout:1657: oid 26 updating version 125 to 126 2024-09-13T13:51:52.408 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 126 (ObjNum 309 snap 44 seq_num 309) dirty exists 2024-09-13T13:51:52.408 INFO:tasks.rados.rados.0.smithi159.stdout:1657: left oid 26 (ObjNum 309 snap 44 seq_num 309) 2024-09-13T13:51:52.408 INFO:tasks.rados.rados.0.smithi159.stdout:1658: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:52.429 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1197543 2024-09-13T13:51:52.434 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:52.433+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:52.455 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1786122 2024-09-13T13:51:52.456 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:51:52.456 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:51:52.457 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:51:52.457 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1660: got (0) Success 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 158 (ObjNum 97 snap 6 seq_num 97) dirty exists 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1657: done (2 left) 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1658: done (1 left) 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1660: done (0 left) 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1661: write (excl) oid 16 current snap is 44 2024-09-13T13:51:52.463 INFO:tasks.rados.rados.0.smithi159.stdout:1661: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-13T13:51:52.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:52.469 INFO:tasks.rados.rados.0.smithi159.stdout:1661: writing smithi15938319-16 from 410170 to 1191480 tid 1 2024-09-13T13:51:52.476 INFO:tasks.rados.rados.0.smithi159.stdout:1661: writing smithi15938319-16 from 1940712 to 2596818 tid 2 2024-09-13T13:51:52.480 INFO:tasks.rados.rados.0.smithi159.stdout:1661: writing smithi15938319-16 from 3163363 to 3210170 tid 3 2024-09-13T13:51:52.480 INFO:tasks.rados.rados.0.smithi159.stdout:1662: snap_create 2024-09-13T13:51:52.480 INFO:tasks.rados.rados.0.smithi159.stdout:1661: finishing write tid 1 to smithi15938319-16 2024-09-13T13:51:52.481 INFO:tasks.rados.rados.0.smithi159.stdout:1661: finishing write tid 2 to smithi15938319-16 2024-09-13T13:51:52.481 INFO:tasks.rados.rados.0.smithi159.stdout:1661: finishing write tid 3 to smithi15938319-16 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: finishing write tid 4 to smithi15938319-16 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: finishing write tid 5 to smithi15938319-16 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: oid 16 updating version 0 to 331 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: oid 16 version 331 is already newer than 330 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: oid 16 version 331 is already newer than 328 2024-09-13T13:51:52.483 INFO:tasks.rados.rados.0.smithi159.stdout:1661: oid 16 version 331 is already newer than 329 2024-09-13T13:51:52.484 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 331 (ObjNum 310 snap 44 seq_num 310) dirty exists 2024-09-13T13:51:52.484 INFO:tasks.rados.rados.0.smithi159.stdout:1661: left oid 16 (ObjNum 310 snap 44 seq_num 310) 2024-09-13T13:51:52.535 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:52.534+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:52.574 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.674934132999994, 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.575 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:51:52.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:51:52.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.674866276000003, 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.582 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:51:52.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:51:52.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344891664000002, 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.588 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:51:52.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:51:52.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.593 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344857188999995, 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.594 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.598 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344842202999999, 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.599 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:52.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344828153999998, 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.604 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:51:52.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344806438999996, 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:51:52.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:51:52.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.344774594, 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:51:52.614 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:51:52.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:52.619 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.183077867999998, 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.621 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:51:52.624 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.183044718000005, 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.625 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:51:52.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.630 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.183014499999999, 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.631 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.632 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.633 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:51:52.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:51:52.635 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.636 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.182972151000001, 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.637 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:52.638 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.639 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:51:52.640 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.182937933999995, 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:51:52.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:51:52.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:51:52.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:51:52.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 84.182904663000002, 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.650 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:51:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:51:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 83.780621429999997, 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.656 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:51:52.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.755554820999997, 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.660 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:51:52.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:51:52.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:51:52.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2587 4.c 4:34327e96:::smithi15938319-44:head [write 1772395~538068] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.749791866000002, 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012151192999999999, 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.667 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2587 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823296+0000", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823297+0000", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4720000000000001e-06 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823897+0000", 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00060006600000000001 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823910+0000", 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2243e-05 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823920+0000", 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0538e-05 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825222+0000", 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001302194 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825304+0000", 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1953999999999995e-05 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.825429+0000", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012415399999999999 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826790+0000", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013615050000000001 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835350+0000", 2024-09-13T13:51:52.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0085598610000000002 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835406+0000", 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5661000000000001e-05 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.835447+0000", 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2964 4.c 4:358f784f:::smithi15938319-41:head [rollback 13] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.699486573999998, 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012071405, 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2964 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873601+0000", 2024-09-13T13:51:52.675 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4699999999999998e-07 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873603+0000", 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.269e-06 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873605+0000", 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7690000000000001e-06 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.676 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873608+0000", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.615e-06 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873725+0000", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000116947 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873727+0000", 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4109999999999999e-06 2024-09-13T13:51:52.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881339+0000", 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007611835 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881439+0000", 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9967e-05 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.881665+0000", 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00022542600000000001 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883193+0000", 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001528263 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885610+0000", 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002417242 2024-09-13T13:51:52.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885643+0000", 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2704999999999999e-05 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.885673+0000", 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9709e-05 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2965 4.c 4:358f784f:::smithi15938319-41:head [append 0~0] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 35.699476298, 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014521253, 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2965 2024-09-13T13:51:52.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873612+0000", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2999999999999999e-07 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873613+0000", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.398e-06 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873616+0000", 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7190000000000001e-06 2024-09-13T13:51:52.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873618+0000", 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6170000000000001e-06 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873745+0000", 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126389 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873747+0000", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4030000000000001e-06 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882806+0000", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090587889999999994 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882861+0000", 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4793999999999997e-05 2024-09-13T13:51:52.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882955+0000", 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4377999999999997e-05 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883725+0000", 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076983499999999996 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888075+0000", 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043496630000000001 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888105+0000", 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0735000000000003e-05 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888133+0000", 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7302999999999999e-05 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.687 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3271 4.c 4:34c00adb:::smithi15938319-12:head [write 537890~789585,stat] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 22.636966237999999, 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.011376862999999999, 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:51:52.688 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3271 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:51:52.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936122+0000", 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.936124+0000", 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4719999999999998e-06 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937704+0000", 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0015798120000000001 2024-09-13T13:51:52.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937743+0000", 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8906000000000003e-05 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937754+0000", 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0885e-05 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937825+0000", 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1216000000000004e-05 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.937830+0000", 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0579999999999997e-06 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940585+0000", 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002755455 2024-09-13T13:51:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940660+0000", 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.940813+0000", 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00015292100000000001 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:52.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947359+0000", 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065464060000000003 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947381+0000", 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2036999999999999e-05 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947459+0000", 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7338999999999999e-05 2024-09-13T13:51:52.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.947498+0000", 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9682999999999999e-05 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:51:52.695 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:51:52.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:52.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:52.698 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:52.697+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:52.798 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:52.797+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:52.813 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:51:52.813 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:51:52.813 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:51:52.813 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:51:52.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:52.870 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:52.870 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:52.892 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:52.892 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:52.898 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:52.898+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:52.953 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:52.954 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:52.954 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:52.957 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:52.957 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:52.966 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:52.966 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:52.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:52.980 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:52.980 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:53.001 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:53.001 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:53.018 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:53.018+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:53.022 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:53.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:53.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:51:53.040 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:53.040+0000 7f4ffdd7e640 -1 osd.1 131 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.040 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:53.039+0000 7f8f5ac65640 -1 osd.6 131 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1661: done (1 left) 2024-09-13T13:51:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1662: done (0 left) 2024-09-13T13:51:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1663: read oid 12 snap -1 2024-09-13T13:51:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1663: expect deleted 2024-09-13T13:51:53.042 INFO:tasks.rados.rados.0.smithi159.stdout:1664: set_chunk oid 47 target oid 2024-09-13T13:51:53.045 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:53.056 INFO:tasks.rados.rados.0.smithi159.stdout:1664: set_chunk oid 47 offset: 619520 length: 1145344 target oid 2050c36c8aa9a35f797993ebbcc3e2f9c7888b1d4f7317e2a2f41bf5535f7ac5 offset: 0 snap -1 2024-09-13T13:51:53.057 INFO:tasks.rados.rados.0.smithi159.stdout:1665: snap_remove snap 39 2024-09-13T13:51:53.068 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:53.068 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:53.079 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.079 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:53.089 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:53.090 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:53.101 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:53.099+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:53.110 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:53.111 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:53.132 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:53.132 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:53.154 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:53.155 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:53.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:53.160+0000 7f4804d65640 0 -- 172.21.15.27:0/4027534861 >> v2:172.21.15.159:6805/4073091144 conn(0x7f4800220470 msgr2=0x7f4800222860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:53.181 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:53.181 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:53.182 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:53.182+0000 7f586a715640 -1 osd.9 131 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:53.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:53.202 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:53.201+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:53.233 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.233 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.241 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:53.303 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:53.303+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:53.346 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.346 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.352 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:53.404 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:53.404+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:53.447 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:53.446+0000 7ffaeece8640 -1 osd.0 131 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.455 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.455 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:53.505 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:53.505+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:53.566 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.566 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:51:53.618 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:53.618+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:53.677 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.677 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.686 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:51:53.707 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:53.706+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:51:53.742 INFO:tasks.rados.rados.0.smithi159.stdout:1665: done (2 left) 2024-09-13T13:51:53.742 INFO:tasks.rados.rados.0.smithi159.stdout:1666: set_chunk oid 29 target oid 2024-09-13T13:51:53.752 INFO:tasks.rados.rados.0.smithi159.stdout:1666: set_chunk oid 29 offset: 527872 length: 412160 target oid 56b297a73fb26a8d27cf7c6d5ef661a450e858439923b6573067af29cc11c9ba offset: 0 snap -1 2024-09-13T13:51:53.753 INFO:tasks.rados.rados.0.smithi159.stdout:1667: write (excl) oid 26 current snap is 45 2024-09-13T13:51:53.753 INFO:tasks.rados.rados.0.smithi159.stdout:1667: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-13T13:51:53.757 INFO:tasks.rados.rados.0.smithi159.stdout:1667: writing smithi15938319-26 from 426977 to 883536 tid 1 2024-09-13T13:51:53.765 INFO:tasks.rados.rados.0.smithi159.stdout:1667: writing smithi15938319-26 from 1482841 to 2282605 tid 2 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1667: writing smithi15938319-26 from 2958080 to 3226977 tid 3 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1668: read oid 4 snap -1 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1668: expect (ObjNum 291 snap 38 seq_num 291) 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1669: delete oid 5 current snap is 45 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1664: finishing set_chunk to oid 47 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 144 (ObjNum 262 snap 30 seq_num 262) dirty exists 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1666: finishing set_chunk to oid 29 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 57 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1667: finishing write tid 1 to smithi15938319-26 2024-09-13T13:51:53.771 INFO:tasks.rados.rados.0.smithi159.stdout:1667: finishing write tid 2 to smithi15938319-26 2024-09-13T13:51:53.772 INFO:tasks.rados.rados.0.smithi159.stdout:1667: finishing write tid 3 to smithi15938319-26 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: finishing write tid 4 to smithi15938319-26 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: finishing write tid 5 to smithi15938319-26 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: oid 26 updating version 0 to 133 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: oid 26 version 133 is already newer than 130 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: oid 26 version 133 is already newer than 131 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: oid 26 version 133 is already newer than 132 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 133 (ObjNum 311 snap 45 seq_num 311) dirty exists 2024-09-13T13:51:53.775 INFO:tasks.rados.rados.0.smithi159.stdout:1667: left oid 26 (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:51:53.782 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:53.781+0000 7f586a715640 -1 osd.9 132 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.783 INFO:tasks.rados.rados.0.smithi159.stdout:1668: expect (ObjNum 291 snap 38 seq_num 291) 2024-09-13T13:51:53.783 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:53.808 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:53.807+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:53.832 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.833 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.835 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:53.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:51:53.880 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:53.879+0000 7f4ffbd7a640 -1 osd.1 132 handle_osd_map injecting map crc failure 2024-09-13T13:51:53.908 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:53.906+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:53.944 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:53.944 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:53.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:51:54.036 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:54.036+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:54.095 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.096 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.104 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:51:54.145 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:54.145+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:54.210 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:54.209+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:54.247 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.247 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.256 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:51:54.312 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:54.311+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:54.370 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.370 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:51:54.413 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:54.411+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:54.482 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.482 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:51:54.521 INFO:tasks.rados.rados.0.smithi159.stdout:1663: done (5 left) 2024-09-13T13:51:54.521 INFO:tasks.rados.rados.0.smithi159.stdout:1664: done (4 left) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1666: done (3 left) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1667: done (2 left) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1668: done (1 left) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1669: done (0 left) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1670: read oid 4 snap 44 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1670: expect (ObjNum 291 snap 38 seq_num 291) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1671: tier_evict oid 43 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1671: tier_evict oid 43 snap -1 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1672: read oid 17 snap -1 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1672: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1673: read oid 33 snap 35 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1673: expect deleted 2024-09-13T13:51:54.522 INFO:tasks.rados.rados.0.smithi159.stdout:1674: write (excl) oid 39 current snap is 45 2024-09-13T13:51:54.523 INFO:tasks.rados.rados.0.smithi159.stdout:1674: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-13T13:51:54.527 INFO:tasks.rados.rados.0.smithi159.stdout:1674: writing smithi15938319-39 from 460591 to 1184001 tid 1 2024-09-13T13:51:54.532 INFO:tasks.rados.rados.0.smithi159.stdout:1674: writing smithi15938319-39 from 1716158 to 2119591 tid 2 2024-09-13T13:51:54.533 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:54.533+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1674: writing smithi15938319-39 from 2729279 to 3169419 tid 3 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1675: tier_promote oid 48 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1676: read oid 9 snap -1 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1676: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1677: delete oid 30 current snap is 45 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1671: got (22) Invalid argument 2024-09-13T13:51:54.539 INFO:tasks.rados.rados.0.smithi159.stdout:1670: expect (ObjNum 291 snap 38 seq_num 291) 2024-09-13T13:51:54.580 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1150516 2024-09-13T13:51:54.597 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.597 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:51:54.619 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:54.647 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:54.646+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:54.670 INFO:tasks.rados.rados.0.smithi159.stdout:1672: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:51:54.715 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:54.713+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.748 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:51:54.811 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1306200 2024-09-13T13:51:54.816 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:54.815+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:54.858 INFO:tasks.rados.rados.0.smithi159.stdout:1674: finishing write tid 1 to smithi15938319-39 2024-09-13T13:51:54.858 INFO:tasks.rados.rados.0.smithi159.stdout:1674: finishing write tid 2 to smithi15938319-39 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1675: got (0) Success 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 297 (ObjNum 296 snap 41 seq_num 296) dirty exists 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1674: finishing write tid 3 to smithi15938319-39 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1670: done (7 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1671: done (6 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1672: done (5 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1673: done (4 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1675: done (3 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1677: done (2 left) 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1678: tier_flush oid 40 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1678: tier_flush oid 40 snap -1 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1674: finishing write tid 4 to smithi15938319-39 2024-09-13T13:51:54.859 INFO:tasks.rados.rados.0.smithi159.stdout:1674: finishing write tid 5 to smithi15938319-39 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: oid 39 updating version 0 to 222 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: oid 39 updating version 222 to 225 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: oid 39 version 225 is already newer than 224 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: oid 39 version 225 is already newer than 223 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 225 (ObjNum 313 snap 45 seq_num 313) dirty exists 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: left oid 39 (ObjNum 313 snap 45 seq_num 313) 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1674: done (2 left) 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1679: read oid 46 snap -1 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1679: expect (ObjNum 247 snap 28 seq_num 247) 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1680: write oid 14 current snap is 45 2024-09-13T13:51:54.860 INFO:tasks.rados.rados.0.smithi159.stdout:1680: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-13T13:51:54.861 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.861 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.862 INFO:tasks.rados.rados.0.smithi159.stdout:1680: writing smithi15938319-14 from 494205 to 968113 tid 1 2024-09-13T13:51:54.867 INFO:tasks.rados.rados.0.smithi159.stdout:1680: writing smithi15938319-14 from 1433122 to 1840224 tid 2 2024-09-13T13:51:54.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:51:54.873 INFO:tasks.rados.rados.0.smithi159.stdout:1680: writing smithi15938319-14 from 2384125 to 2846809 tid 3 2024-09-13T13:51:54.873 INFO:tasks.rados.rados.0.smithi159.stdout:1681: read oid 28 snap 40 2024-09-13T13:51:54.873 INFO:tasks.rados.rados.0.smithi159.stdout:1681: expect deleted 2024-09-13T13:51:54.873 INFO:tasks.rados.rados.0.smithi159.stdout:1682: set_chunk oid 41 target oid 2024-09-13T13:51:54.877 INFO:tasks.rados.rados.0.smithi159.stdout:1682: set_chunk oid 41 offset: 1991680 length: 39424 target oid 316b955df0fe85e65e5bfb4a5d6ad33d4cb34a6e1735da7aa03be5f90c7c02e2 offset: 0 snap 42 2024-09-13T13:51:54.877 INFO:tasks.rados.rados.0.smithi159.stdout:1683: set_chunk oid 26 target oid 2024-09-13T13:51:54.897 INFO:tasks.rados.rados.0.smithi159.stdout:1683: set_chunk oid 26 offset: 344064 length: 2749952 target oid 24df0bd1f63206923d0fd861983b92c8a7695d807e2db43bda25d9432b3a087f offset: 0 snap -1 2024-09-13T13:51:54.897 INFO:tasks.rados.rados.0.smithi159.stdout:1684: read oid 43 snap 28 2024-09-13T13:51:54.897 INFO:tasks.rados.rados.0.smithi159.stdout:1684: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:54.897 INFO:tasks.rados.rados.0.smithi159.stdout:1685: snap_remove snap 44 2024-09-13T13:51:54.917 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:54.915+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:54.972 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:54.972 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:54.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:51:55.017 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:55.016+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:55.046 INFO:tasks.rados.rados.0.smithi159.stdout:1685: done (8 left) 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1686: read oid 27 snap -1 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1686: expect (ObjNum 223 snap 24 seq_num 223) 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1687: read oid 1 snap -1 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1687: expect (ObjNum 287 snap 38 seq_num 287) 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1688: read oid 38 snap -1 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1688: expect (ObjNum 303 snap 43 seq_num 303) 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1689: read oid 45 snap -1 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1689: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:51:55.047 INFO:tasks.rados.rados.0.smithi159.stdout:1690: set_chunk oid 17 target oid 2024-09-13T13:51:55.050 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:55.050+0000 7f1a506ea640 -1 osd.13 133 handle_osd_map injecting map crc failure 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1690: set_chunk oid 17 offset: 1454592 length: 267776 target oid ac744b71e4d74ce94cc0cc4a7a4028089f68bfbb27ed623cf2677a503a731b67 offset: 0 snap -1 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1691: read oid 35 snap 34 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1691: expect (ObjNum 267 snap 32 seq_num 267) 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1692: read oid 25 snap -1 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1692: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1693: read oid 50 snap 42 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1693: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1678: got (16) Device or resource busy 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 40 v 170 (ObjNum 175 snap 16 seq_num 175) unflushed 32 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1678: done (15 left) 2024-09-13T13:51:55.054 INFO:tasks.rados.rados.0.smithi159.stdout:1694: tier_evict oid 7 2024-09-13T13:51:55.055 INFO:tasks.rados.rados.0.smithi159.stdout:1694: tier_evict oid 7 snap -1 2024-09-13T13:51:55.055 INFO:tasks.rados.rados.0.smithi159.stdout:1679: expect (ObjNum 247 snap 28 seq_num 247) 2024-09-13T13:51:55.057 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:55.057+0000 7f8f58c61640 -1 osd.6 133 handle_osd_map injecting map crc failure 2024-09-13T13:51:55.069 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:55.092 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:55.093 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:55.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:51:55.118 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:55.117+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: finishing write tid 1 to smithi15938319-14 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: finishing write tid 2 to smithi15938319-14 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: finishing write tid 3 to smithi15938319-14 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: finishing write tid 4 to smithi15938319-14 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: finishing write tid 5 to smithi15938319-14 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: oid 14 updating version 0 to 202 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: oid 14 updating version 202 to 204 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: oid 14 version 204 is already newer than 201 2024-09-13T13:51:55.124 INFO:tasks.rados.rados.0.smithi159.stdout:1680: oid 14 version 204 is already newer than 203 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 204 (ObjNum 315 snap 45 seq_num 315) dirty exists 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:1680: left oid 14 (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:1682: finishing set_chunk to oid 41 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 41 set_chunk returned error code -95 offset: 1991680 length: 39424 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 0 (ObjNum 255 snap 30 seq_num 255) dirty exists 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:1683: finishing set_chunk to oid 26 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 133 (ObjNum 311 snap 45 seq_num 311) dirty exists 2024-09-13T13:51:55.125 INFO:tasks.rados.rados.0.smithi159.stdout:1684: expect (ObjNum 195 snap 18 seq_num 195) 2024-09-13T13:51:55.129 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1540004 2024-09-13T13:51:55.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:51:55.205 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:55.205 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:55.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:51:55.224 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3254539 2024-09-13T13:51:55.258 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:55.257+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:55.289 INFO:tasks.rados.rados.0.smithi159.stdout:1676: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:51:55.316 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:55.316 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:55.320 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:55.318+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:55.325 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:51:55.334 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:55.360 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:55.378 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:55.378 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:55.382 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:55.399 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:55.400 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:55.400 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:55.420 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:55.418+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:55.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:55.432 INFO:tasks.rados.rados.0.smithi159.stdout:1687: expect (ObjNum 287 snap 38 seq_num 287) 2024-09-13T13:51:55.442 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:55.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:55.467 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:55.467 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:55.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:51:55.485 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:55.490 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2085846 2024-09-13T13:51:55.507 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:55.521 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:55.519+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:55.528 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:55.528 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:55.546 INFO:tasks.rados.rados.0.smithi159.stdout:1689: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:51:55.549 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:55.549 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:55.569 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:55.570 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:55.579 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:55.579 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:55.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:51:55.591 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:55.609 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2402647 2024-09-13T13:51:55.613 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:55.613 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:55.618+0000 7f50a1ae6640 0 -- 172.21.15.27:0/673670330 >> v2:172.21.15.159:6805/4073091144 conn(0x7f509c220430 msgr2=0x7f509c222820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:55.622 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:55.621+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:55.640 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:55.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:55.662 INFO:tasks.rados.rados.0.smithi159.stdout:1688: expect (ObjNum 303 snap 43 seq_num 303) 2024-09-13T13:51:55.662 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.693 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.46375693, 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:51:55.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:51:55.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:51:55.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.700 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.95901874800001, 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:51:55.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.706 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.958969436, 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:51:55.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:51:55.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.958953732, 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:51:55.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:51:55.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.958926056, 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.721 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:51:55.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.92370845799999, 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:51:55.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:51:55.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:51:55.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.916639476, 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:51:55.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.882101014, 2024-09-13T13:51:55.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:51:55.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:51:55.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.746 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.874471172, 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:51:55.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:51:55.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:51:55.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.753 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.87109479999999, 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:51:55.754 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:55.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:51:55.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:55.760 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.76823552, 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:51:55.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.766 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 98.557211980000005, 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 98.545696696999997, 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:51:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:55.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:51:55.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.444388228999998, 2024-09-13T13:51:55.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:55.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:51:55.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.444323881000003, 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.781 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:51:55.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:51:55.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.444295819000004, 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:51:55.787 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:55.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:51:55.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:51:55.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.442613929000004, 2024-09-13T13:51:55.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:51:55.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:51:55.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.003085721000005, 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:51:55.801 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:51:55.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.874449966, 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:51:55.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:51:55.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:51:55.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.729193366000001, 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:55.811 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:51:55.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.814 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:51:55.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:51:55.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:51:55.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:51:55.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:51:55.821 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:51:55.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:51:55.823 INFO:tasks.rados.rados.0.smithi159.stdout:1691: expect (ObjNum 267 snap 32 seq_num 267) 2024-09-13T13:51:55.824 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:55.821+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:55.824 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:51:55.898 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:55.923 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:55.921+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:55.929 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:55.929 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:55.929 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:55.929 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:55.938 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:51:55.968 INFO:tasks.rados.rados.0.smithi159.stdout:1690: finishing set_chunk to oid 17 2024-09-13T13:51:55.969 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 292 (ObjNum 292 snap 41 seq_num 292) dirty exists 2024-09-13T13:51:55.969 INFO:tasks.rados.rados.0.smithi159.stdout:1693: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:51:55.969 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:55.997 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.025 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:56.024+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:56.034 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1694292 2024-09-13T13:51:56.042 INFO:tasks.rados.rados.0.smithi159.stdout:1692: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:51:56.042 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.074 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:56.074 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:56.074 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:56.074 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:56.074 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:56.082 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:51:56.125 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:56.123+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:56.138 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 379075 2024-09-13T13:51:56.229 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:56.225+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:56.230 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:56.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.49502511999999, 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.231 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:51:56.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.235 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:51:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.237 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.494947719, 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.238 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:51:56.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:51:56.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:51:56.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.244 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.494795181, 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.246 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:51:56.247 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:51:56.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:51:56.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.494689142, 2024-09-13T13:51:56.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:56.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:51:56.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:51:56.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.257 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.48635006400001, 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.262 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.458983354, 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.263 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:51:56.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:51:56.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.268 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.45721675599999, 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.269 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:51:56.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:51:56.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.275 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.452337414, 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:51:56.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:51:56.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:51:56.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.441825689, 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.282 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.287 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:51:56.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.417568399, 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.290 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.292 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:51:56.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.414683354, 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.296 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:51:56.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.301 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:56.302 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.41247932900001, 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:51:56.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.306 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.40548868499999, 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:51:56.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.313 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.300497071, 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:51:56.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:56.319 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.476907209999993, 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:56.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:51:56.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.325 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.476874420000001, 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.326 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:51:56.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:51:56.331 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.361114912999994, 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:51:56.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.334 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.335 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.436512223999998, 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.336 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:51:56.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:56.339 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:51:56.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.342 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.486302477000002, 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:51:56.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3286 3.2 e114/73 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head v 114'133, mlcod=105'131)", 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:51:56.347 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 25.259333184999999, 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74551292800000002, 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.73e-07 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.968632+0000", 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075455699999999999 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.349 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711912+0000", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74328065200000004 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711933+0000", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1279000000000001e-05 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712080+0000", 2024-09-13T13:51:56.350 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014671199999999999 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712101+0000", 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0339999999999998e-05 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713357+0000", 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012564550000000001 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.351 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713390+0000", 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2759999999999998e-05 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.352 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:56.353 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:56.353+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:56.354 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:51:56.355 INFO:tasks.rados.rados.0.smithi159.stdout:1686: expect (ObjNum 223 snap 24 seq_num 223) 2024-09-13T13:51:56.355 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.366 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.409 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 542241 2024-09-13T13:51:56.447 INFO:tasks.rados.rados.0.smithi159.stdout:1694: got (22) Invalid argument 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1676: done (15 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1679: done (14 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1680: done (13 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1681: done (12 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1682: done (11 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1683: done (10 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1684: done (9 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1686: done (8 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1687: done (7 left) 2024-09-13T13:51:56.448 INFO:tasks.rados.rados.0.smithi159.stdout:1688: done (6 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1689: done (5 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1690: done (4 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1691: done (3 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1692: done (2 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1693: done (1 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1694: done (0 left) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1695: read oid 40 snap -1 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1695: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1696: read oid 15 snap 37 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1696: expect deleted 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1697: read oid 22 snap -1 2024-09-13T13:51:56.449 INFO:tasks.rados.rados.0.smithi159.stdout:1697: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:51:56.450 INFO:tasks.rados.rados.0.smithi159.stdout:1698: delete oid 16 current snap is 45 2024-09-13T13:51:56.450 INFO:tasks.rados.rados.0.smithi159.stdout:1696: done (3 left) 2024-09-13T13:51:56.450 INFO:tasks.rados.rados.0.smithi159.stdout:1698: done (2 left) 2024-09-13T13:51:56.450 INFO:tasks.rados.rados.0.smithi159.stdout:1699: tier_promote oid 21 2024-09-13T13:51:56.450 INFO:tasks.rados.rados.0.smithi159.stdout:1700: snap_create 2024-09-13T13:51:56.454 INFO:tasks.rados.rados.0.smithi159.stdout:1699: got (0) Success 2024-09-13T13:51:56.455 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 147 (ObjNum 305 snap 43 seq_num 305) dirty exists 2024-09-13T13:51:56.455 INFO:tasks.rados.rados.0.smithi159.stdout:1697: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:51:56.455 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.455 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:56.454+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:56.476 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:56.476 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:56.476 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:56.476 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:56.485 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:51:56.517 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.561 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:56.559+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:56.621 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:56.622 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:56.622 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:56.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:51:56.642 INFO:tasks.rados.rados.0.smithi159.stdout:1695: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:51:56.656 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:56.655+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:56.696 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 220657 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:212 3.0 e41/33 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:56.748 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 104.00410540599999, 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0086005760000000004, 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741315+0000", 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742745+0000", 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001429835 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742795+0000", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0194000000000001e-05 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742808+0000", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.295e-05 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742883+0000", 2024-09-13T13:51:56.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999995e-05 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.742910+0000", 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6282999999999999e-05 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749856+0000", 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0069467319999999997 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749915+0000", 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8746000000000003e-05 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.998524147, 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.753 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.757 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.995547536, 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.758 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.759 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.760 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:51:56.761 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.762 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.993573365, 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:56.763 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.764 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.765 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:51:56.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:226 3.0 e41/33 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'6, mlcod=41'4)", 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.959814264, 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0081958840000000005, 2024-09-13T13:51:56.768 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785606+0000", 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7000000000000004e-07 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786301+0000", 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00069499099999999995 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786339+0000", 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7588e-05 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786346+0000", 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786398+0000", 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2178000000000001e-05 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.786427+0000", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8603999999999999e-05 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793750+0000", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073231600000000004 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.793802+0000", 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1535e-05 2024-09-13T13:51:56.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.95484284200001, 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:51:56.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:51:56.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:51:56.777 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.950941551, 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:51:56.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.910903483, 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:51:56.782 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:51:56.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.786 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:239 3.0 e41/33 3:1e720c31:::smithi15938319-40 low_tier:head v 41'13, mlcod=41'9)", 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.905727641, 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082916250000000004, 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839692+0000", 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839694+0000", 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.437e-06 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840509+0000", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081506600000000003 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840559+0000", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0652000000000002e-05 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840572+0000", 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2986000000000001e-05 2024-09-13T13:51:56.789 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840645+0000", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2978000000000005e-05 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.840669+0000", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3810000000000001e-05 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847944+0000", 2024-09-13T13:51:56.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072750039999999998 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.847984+0000", 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9691999999999999e-05 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:56.791 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.829776563, 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:51:56.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:51:56.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:56.797 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.82497566799999, 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:56.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:51:56.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 102.670336831, 2024-09-13T13:51:56.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.804 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:51:56.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:817 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:56.809 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 98.547455678000006, 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082900909999999994, 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 817 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.810 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197964+0000", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197967+0000", 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197978+0000", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1158000000000001e-05 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198003+0000", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4070000000000002e-05 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198017+0000", 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4018e-05 2024-09-13T13:51:56.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198104+0000", 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7267000000000003e-05 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198203+0000", 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9563999999999997e-05 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.204870+0000", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006666707 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.205785+0000", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511100000000003 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206184+0000", 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039845499999999998 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206219+0000", 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4993999999999997e-05 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.206254+0000", 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5756000000000001e-05 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.815 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 88.490475344999993, 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:51:56.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:51:56.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:51:56.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1253 4.6 4:6668b762:::smithi15938319-26:head [write 1866533~496024] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 88.485162496000001, 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0087663600000000008, 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1253 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260257+0000", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260259+0000", 2024-09-13T13:51:56.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.545e-06 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260491+0000", 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00023235899999999999 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260523+0000", 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1659000000000003e-05 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.824 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260531+0000", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7919999999999993e-06 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260584+0000", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3406999999999999e-05 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260658+0000", 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3949000000000006e-05 2024-09-13T13:51:56.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.260806+0000", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014805800000000001 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.265304+0000", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0044975299999999996 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268929+0000", 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036255929999999999 2024-09-13T13:51:56.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268982+0000", 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2808e-05 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.269024+0000", 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1659999999999998e-05 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 75.457383988999993, 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:51:56.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:56.831 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.948743733999997, 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:51:56.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:51:56.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 34.834716473999997, 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.836 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:51:56.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:51:56.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:51:56.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:56.845 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 33.837114114999999, 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:56.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:51:56.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:56.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3760 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e131)", 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:52.038065+0000", 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 4.7073554590000004, 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.008183737, 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:56.852 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3760 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038065+0000", 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038065+0000", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038065+0000", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2699999999999999e-07 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038068+0000", 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2140000000000001e-06 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038074+0000", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0689999999999998e-06 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038080+0000", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6289999999999998e-06 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038125+0000", 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4821999999999999e-05 2024-09-13T13:51:56.855 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.038128+0000", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9900000000000002e-06 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.044987+0000", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006859179 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.045037+0000", 2024-09-13T13:51:56.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9818999999999999e-05 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:52.046248+0000", 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012114879999999999 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:56.857 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:56.858 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:56.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:51:56.860 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:56.858+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:56.959 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:56.959+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:57.003 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.004 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:57.004 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:57.004 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.012 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:51:57.049 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:51:57.052 INFO:tasks.rados.rados.0.smithi159.stdout:1695: done (3 left) 2024-09-13T13:51:57.052 INFO:tasks.rados.rados.0.smithi159.stdout:1697: done (2 left) 2024-09-13T13:51:57.052 INFO:tasks.rados.rados.0.smithi159.stdout:1699: done (1 left) 2024-09-13T13:51:57.052 INFO:tasks.rados.rados.0.smithi159.stdout:1700: done (0 left) 2024-09-13T13:51:57.053 INFO:tasks.rados.rados.0.smithi159.stdout:1701: read oid 6 snap -1 2024-09-13T13:51:57.053 INFO:tasks.rados.rados.0.smithi159.stdout:1701: expect (ObjNum 274 snap 37 seq_num 274) 2024-09-13T13:51:57.053 INFO:tasks.rados.rados.0.smithi159.stdout:1702: write oid 8 current snap is 46 2024-09-13T13:51:57.053 INFO:tasks.rados.rados.0.smithi159.stdout:1702: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-13T13:51:57.058 INFO:tasks.rados.rados.0.smithi159.stdout:1702: writing smithi15938319-8 from 527819 to 1152225 tid 1 2024-09-13T13:51:57.061 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:57.060+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:57.065 INFO:tasks.rados.rados.0.smithi159.stdout:1702: writing smithi15938319-8 from 1833733 to 2244504 tid 2 2024-09-13T13:51:57.072 INFO:tasks.rados.rados.0.smithi159.stdout:1702: writing smithi15938319-8 from 2722618 to 3324199 tid 3 2024-09-13T13:51:57.072 INFO:tasks.rados.rados.0.smithi159.stdout:1703: rollback oid 34 current snap is 46 2024-09-13T13:51:57.072 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 34 to 42 2024-09-13T13:51:57.072 INFO:tasks.rados.rados.0.smithi159.stdout:1701: expect (ObjNum 274 snap 37 seq_num 274) 2024-09-13T13:51:57.072 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:57.130 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.130 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:57.130 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:57.130 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:57.130 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.139 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:51:57.147 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1092153 2024-09-13T13:51:57.191 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:57.190+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:57.240 INFO:tasks.rados.rados.0.smithi159.stdout:1702: finishing write tid 1 to smithi15938319-8 2024-09-13T13:51:57.240 INFO:tasks.rados.rados.0.smithi159.stdout:1702: finishing write tid 2 to smithi15938319-8 2024-09-13T13:51:57.240 INFO:tasks.rados.rados.0.smithi159.stdout:1701: done (2 left) 2024-09-13T13:51:57.240 INFO:tasks.rados.rados.0.smithi159.stdout:1704: set_chunk oid 39 target oid 2024-09-13T13:51:57.253 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:57.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 104.990130763, 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.254 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:51:57.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 104.98917726400001, 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:51:57.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:126 4.8 e41/34 4:165e03eb:::smithi15938319-32:head v 41'5, mlcod=41'3)", 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 104.76292728999999, 2024-09-13T13:51:57.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036490759999999998, 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488939+0000", 2024-09-13T13:51:57.264 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.488940+0000", 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9400000000000004e-07 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489065+0000", 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000125652 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.265 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489092+0000", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6506e-05 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489097+0000", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9119999999999997e-06 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489145+0000", 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8059000000000002e-05 2024-09-13T13:51:57.266 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.489166+0000", 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1226999999999999e-05 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492548+0000", 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0033817610000000001 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.267 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.492588+0000", 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0164999999999999e-05 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 93.478341004000001, 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.268 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.269 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:51:57.270 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.271 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.272 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1218 4.3 e52/34 4:c1ae59bf:::smithi15938319-45:head v 52'19, mlcod=52'17)", 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.032975223999998, 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039766309999999996, 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.273 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218891+0000", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.218892+0000", 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9400000000000005e-07 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.274 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219452+0000", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00055999700000000001 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219474+0000", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2470999999999999e-05 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219500+0000", 2024-09-13T13:51:57.275 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5449e-05 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219540+0000", 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0948999999999997e-05 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.219553+0000", 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.207e-05 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.276 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222830+0000", 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032778920000000001 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.222868+0000", 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7109e-05 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:51:57.277 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.925430430999995, 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.278 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.279 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.280 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.281 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.924681542000002, 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.282 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.283 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.284 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.285 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:51:57.286 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.506347360000007, 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.287 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:51:57.288 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.289 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:51:57.290 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.291 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.501219001999999, 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.292 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:51:57.293 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.294 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.295 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.296 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 66.500307445999994, 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.297 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:51:57.298 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.299 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:51:57.300 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.472221357999999, 2024-09-13T13:51:57.301 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:57.302 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.303 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.304 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:51:57.305 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:138 4.8 e80/34 4:1b24d731:::smithi15938319-13:11 v 80'75, mlcod=78'69)", 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.471555033999998, 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038853730000000001, 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.306 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780311+0000", 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.307 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780312+0000", 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4200000000000004e-07 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780322+0000", 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0383999999999999e-05 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780367+0000", 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4783000000000001e-05 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.308 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780378+0000", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0292000000000001e-05 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780701+0000", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00032313399999999999 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.780722+0000", 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1353e-05 2024-09-13T13:51:57.309 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784153+0000", 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003430986 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.784196+0000", 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3498999999999998e-05 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.310 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2559 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'82, mlcod=84'79)", 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.406321063, 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040899279999999996, 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.311 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845546+0000", 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.312 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845704+0000", 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015839000000000001 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845732+0000", 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7736e-05 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845738+0000", 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.313 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845791+0000", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845807+0000", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5755000000000001e-05 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849592+0000", 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037853499999999998 2024-09-13T13:51:57.314 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849635+0000", 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2669000000000001e-05 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.402947013999999, 2024-09-13T13:51:57.315 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:51:57.316 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.317 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.318 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:51:57.319 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.402178966000001, 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.320 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:51:57.321 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.322 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.323 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.324 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 50.045865685999999, 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.325 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:51:57.326 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.327 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:51:57.328 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3424 4.8 e118/34 4:1b24d731:::smithi15938319-13:head v 118'122, mlcod=115'119)", 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.269258462, 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041392160000000002, 2024-09-13T13:51:57.329 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:57.330 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.34e-07 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983037+0000", 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042880899999999999 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.331 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983075+0000", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7994000000000002e-05 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983083+0000", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9030000000000004e-06 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983136+0000", 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3353999999999998e-05 2024-09-13T13:51:57.332 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983152+0000", 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5583999999999999e-05 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986717+0000", 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035656059999999998 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.333 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986747+0000", 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9532000000000001e-05 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 19.838458420999999, 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.334 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.335 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:51:57.336 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.337 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.338 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.945773442, 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.339 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:51:57.340 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.341 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:51:57.342 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.343 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.294181791, 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.344 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.345 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:51:57.346 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.347 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.348 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.350 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:57.295+0000 7fe0fbe07640 -1 osd.2 134 handle_osd_map injecting map crc failure 2024-09-13T13:51:57.350 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:57.349+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:51:57.351 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1704: set_chunk oid 39 offset: 113664 length: 3041792 target oid 47384ab1252e7b643113c6239100055f3d4ef81ce08924633dad99e0fab7bbd9 offset: 0 snap -1 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1705: write oid 2 current snap is 46 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1702: finishing write tid 3 to smithi15938319-8 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1705: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1705: writing smithi15938319-2 from 544626 to 1244281 tid 1 2024-09-13T13:51:57.353 INFO:tasks.rados.rados.0.smithi159.stdout:1705: writing smithi15938319-2 from 1892215 to 2446644 tid 2 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1705: writing smithi15938319-2 from 3150041 to 3344626 tid 3 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1706: tier_flush oid 49 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1706: tier_flush oid 49 snap -1 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: finishing write tid 4 to smithi15938319-8 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: finishing write tid 5 to smithi15938319-8 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: oid 8 updating version 0 to 157 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: oid 8 version 157 is already newer than 156 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: oid 8 updating version 157 to 159 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: oid 8 version 159 is already newer than 158 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 159 (ObjNum 317 snap 46 seq_num 317) dirty exists 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1702: left oid 8 (ObjNum 317 snap 46 seq_num 317) 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1703: finishing rollback tid 0 to smithi15938319-34 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1703: finishing rollback tid 1 to smithi15938319-34 2024-09-13T13:51:57.354 INFO:tasks.rados.rados.0.smithi159.stdout:1703: finishing rollback tid 2 to smithi15938319-34 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 227 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1702: done (4 left) 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1703: done (3 left) 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1707: write (excl) oid 48 current snap is 46 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1707: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1707: writing smithi15938319-48 from 561433 to 1336337 tid 1 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1707: writing smithi15938319-48 from 1950697 to 2365137 tid 2 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1707: writing smithi15938319-48 from 2777464 to 3361433 tid 3 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1708: write oid 4 current snap is 46 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1708: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1708: writing smithi15938319-4 from 578240 to 1144746 tid 1 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1708: writing smithi15938319-4 from 1609179 to 2167277 tid 2 2024-09-13T13:51:57.355 INFO:tasks.rados.rados.0.smithi159.stdout:1708: writing smithi15938319-4 from 2804887 to 3356637 tid 3 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1709: write oid 42 current snap is 46 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1709: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1709: writing smithi15938319-42 from 595047 to 1236802 tid 1 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1709: writing smithi15938319-42 from 1667661 to 2085770 tid 2 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1709: writing smithi15938319-42 from 2548663 to 3311685 tid 3 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1710: write oid 12 current snap is 46 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1710: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1710: writing smithi15938319-12 from 611854 to 1328858 tid 1 2024-09-13T13:51:57.356 INFO:tasks.rados.rados.0.smithi159.stdout:1710: writing smithi15938319-12 from 2126143 to 2687910 tid 2 2024-09-13T13:51:57.359 INFO:tasks.rados.rados.0.smithi159.stdout:1710: writing smithi15938319-12 from 3259733 to 3411854 tid 3 2024-09-13T13:51:57.359 INFO:tasks.rados.rados.0.smithi159.stdout:1711: snap_remove snap 42 2024-09-13T13:51:57.431 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:57.431+0000 7ffaeece8640 -1 osd.0 134 handle_osd_map injecting map crc failure 2024-09-13T13:51:57.451 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:57.450+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:57.460 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.460 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:57.460 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:51:57.460 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.469 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:51:57.553 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:57.552+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:57.611 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.611 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:51:57.611 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:51:57.611 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:51:57.611 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:51:57.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:51:57.660 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:57.659+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:57.753 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:57.753+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:57.765 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:51:57.765 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:51:57.765 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.221375405, 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.766 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.767 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:51:57.768 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.769 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.770 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:51:57.771 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 127.56193333900001, 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:51:57.772 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.773 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:51:57.774 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.775 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.776 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.777 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 127.560545115, 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.778 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.779 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:51:57.780 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.781 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:51:57.782 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:121 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'91, mlcod=26'83)", 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 127.160116972, 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0057298169999999999, 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.783 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.784 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603273+0000", 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9800000000000004e-07 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603277+0000", 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9099999999999998e-06 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.785 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603296+0000", 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8382000000000001e-05 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603301+0000", 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9570000000000001e-06 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.786 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603340+0000", 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9604999999999998e-05 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.603356+0000", 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5503999999999999e-05 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.787 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.608973+0000", 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005617113 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.609003+0000", 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9848e-05 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.788 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 104.846559816, 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.789 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.790 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.791 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:51:57.792 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.793 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.794 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.795 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.796 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 103.99272106799999, 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.797 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.798 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.799 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.800 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:51:57.801 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.802 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.803 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 99.573286198999995, 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.804 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:57.805 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:51:57.806 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.807 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:51:57.808 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:51:57.809 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 99.573238248999999, 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.810 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.811 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.812 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:51:57.813 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.814 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.815 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 97.884775321999996, 2024-09-13T13:51:57.816 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.817 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.818 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.819 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:51:57.820 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.821 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.822 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.823 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.189168194999993, 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.824 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.825 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:51:57.826 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.827 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.828 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:51:57.829 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.830 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.461465920999999, 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:51:57.831 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.832 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.833 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.834 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.835 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:51:57.836 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2056 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.435956516000005, 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0055690399999999999, 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.837 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2056 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.838 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327433+0000", 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327435+0000", 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.854e-06 2024-09-13T13:51:57.839 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327444+0000", 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4800000000000001e-06 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327461+0000", 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7725000000000001e-05 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.840 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327470+0000", 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6950000000000006e-06 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327536+0000", 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6184000000000002e-05 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.841 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327618+0000", 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1278000000000003e-05 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333002+0000", 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053848239999999999 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.842 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.435290778999999, 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.843 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.844 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.845 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.846 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:51:57.847 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.848 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.435249763000002, 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.849 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.850 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.851 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:51:57.852 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.853 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.854 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.616112854999997, 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.855 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.856 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.857 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.858 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.859 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:51:57.860 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.861 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3275 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:57.862 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.821133761999999, 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061117630000000001, 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3275 2024-09-13T13:51:57.863 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.864 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942258+0000", 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9369999999999998e-06 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942265+0000", 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-06 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.865 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942282+0000", 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6880000000000001e-05 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942292+0000", 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8689999999999999e-06 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7336000000000005e-05 2024-09-13T13:51:57.866 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942388+0000", 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3090000000000001e-06 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945082+0000", 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026939160000000002 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.867 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945163+0000", 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0728000000000004e-05 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945384+0000", 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221375 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.868 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948177+0000", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027923119999999999 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948231+0000", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948299+0000", 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8300000000000007e-05 2024-09-13T13:51:57.869 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948368+0000", 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8872000000000001e-05 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3276 4.9 4:977d569f:::smithi15938319-22:head [rollback 12] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.821069276999999, 2024-09-13T13:51:57.870 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082025069999999995, 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3276 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.871 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-09-13T13:51:57.872 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942325+0000", 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.512e-06 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942335+0000", 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0016e-05 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.873 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942341+0000", 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942471+0000", 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013015300000000001 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:57.874 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942478+0000", 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.55e-06 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946541+0000", 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040631779999999998 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946640+0000", 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8694000000000001e-05 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.875 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946923+0000", 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028351899999999998 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948388+0000", 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001464414 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.876 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950419+0000", 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020313689999999999 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950465+0000", 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5519999999999998e-05 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.877 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950523+0000", 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.842e-05 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3277 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 27.821010955999999, 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010801006, 2024-09-13T13:51:57.878 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3277 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.879 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942380+0000", 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.880 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942384+0000", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1949999999999996e-06 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942391+0000", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0339999999999999e-06 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942396+0000", 2024-09-13T13:51:57.881 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8409999999999999e-06 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942508+0000", 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112458 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942512+0000", 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.579e-06 2024-09-13T13:51:57.882 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947640+0000", 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051287859999999998 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947737+0000", 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6329999999999994e-05 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.883 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947934+0000", 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019687900000000001 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948478+0000", 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000544448 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.884 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953066+0000", 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00458831 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953129+0000", 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2280999999999995e-05 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.885 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953180+0000", 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3520 4.9 4:977d569f:::smithi15938319-22:head [read 627200~1132544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.788807824999999, 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062848280000000001, 2024-09-13T13:51:57.886 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3520 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:57.887 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974583+0000", 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-09-13T13:51:57.888 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974589+0000", 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4879999999999998e-06 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974600+0000", 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1232e-05 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.889 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974609+0000", 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3430000000000002e-06 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974667+0000", 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7388e-05 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:51:57.890 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974672+0000", 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6930000000000004e-06 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979695+0000", 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050231449999999997 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979758+0000", 2024-09-13T13:51:57.891 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2232999999999999e-05 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.980867+0000", 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001109229 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.892 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3660 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 2c=[2c,2b,2a,29,28,27,26,25,24,23,22,1e,1a,14,12] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.060065519, 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068015810000000001, 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3660 2024-09-13T13:51:57.893 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:51:57.894 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703325+0000", 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.91e-07 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703326+0000", 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6759999999999999e-06 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.895 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703331+0000", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6879999999999998e-06 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703335+0000", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8940000000000003e-06 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706175+0000", 2024-09-13T13:51:57.896 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028405430000000001 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706229+0000", 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3813000000000001e-05 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706343+0000", 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114123 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.897 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706835+0000", 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049211600000000002 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710024+0000", 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003188116 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.898 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710080+0000", 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6168999999999997e-05 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710126+0000", 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6152000000000001e-05 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:51:57.899 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:51:57.900 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:51:57.901 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:51:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:51:57.902 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:51:57.902 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:51:57.903 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:51:57.903 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:51:57.903 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:51:57.904 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:51:57.904 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:51:57.904 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:51:57.905 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:51:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:51:57.905 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:51:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:51:57.906 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:57.899+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:57.923 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:51:57.924 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:51:57.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:51:57.944 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:51:57.966 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:51:57.966 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:51:57.988 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:51:57.988 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:51:58.001 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:51:58.000+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:51:58.009 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:51:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:51:58.013 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.013 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.022 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:51:58.030 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:51:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:51:58.052 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:51:58.053 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:51:58.054 INFO:tasks.rados.rados.0.smithi159.stdout:1711: done (7 left) 2024-09-13T13:51:58.054 INFO:tasks.rados.rados.0.smithi159.stdout:1712: write (excl) oid 34 current snap is 46 2024-09-13T13:51:58.054 INFO:tasks.rados.rados.0.smithi159.stdout:1712: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-13T13:51:58.061 INFO:tasks.rados.rados.0.smithi159.stdout:1712: writing smithi15938319-34 from 628661 to 1420914 tid 1 2024-09-13T13:51:58.062 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:58.062+0000 7faeac214640 -1 osd.8 135 handle_osd_map injecting map crc failure 2024-09-13T13:51:58.068 INFO:tasks.rados.rados.0.smithi159.stdout:1712: writing smithi15938319-34 from 2184625 to 2606403 tid 2 2024-09-13T13:51:58.071 INFO:tasks.rados.rados.0.smithi159.stdout:1712: writing smithi15938319-34 from 3403509 to 3428661 tid 3 2024-09-13T13:51:58.071 INFO:tasks.rados.rados.0.smithi159.stdout:1713: rollback oid 44 current snap is 46 2024-09-13T13:51:58.071 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 44 to 35 2024-09-13T13:51:58.071 INFO:tasks.rados.rados.0.smithi159.stdout:1714: read oid 36 snap -1 2024-09-13T13:51:58.072 INFO:tasks.rados.rados.0.smithi159.stdout:1714: expect (ObjNum 280 snap 38 seq_num 280) 2024-09-13T13:51:58.072 INFO:tasks.rados.rados.0.smithi159.stdout:1715: set_chunk oid 29 target oid 2024-09-13T13:51:58.072 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:51:58.071+0000 7fe0fbe07640 -1 osd.2 135 handle_osd_map injecting map crc failure 2024-09-13T13:51:58.079 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:51:58.079 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:51:58.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:51:58.079+0000 7f552d07f640 0 -- 172.21.15.27:0/3301370265 >> v2:172.21.15.159:6805/4073091144 conn(0x7f55282223d0 msgr2=0x7f55282247c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1715: set_chunk oid 29 offset: 2386944 length: 836096 target oid 1843e5017155328298af732ab71c733948e3e2430ef22524188c9967ff588f1a offset: 0 snap -1 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1716: rollback oid 15 current snap is 46 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 15 to 32 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:1704: finishing set_chunk to oid 39 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 113664 length: 3041792 2024-09-13T13:51:58.090 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 0 (ObjNum 313 snap 45 seq_num 313) dirty exists 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: finishing write tid 1 to smithi15938319-2 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1704: done (11 left) 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1717: read oid 8 snap 32 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1717: expect (ObjNum 261 snap 30 seq_num 261) 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1718: tier_promote oid 17 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: finishing write tid 2 to smithi15938319-2 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1706: got (16) Device or resource busy 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 49 v 55 (ObjNum 173 snap 16 seq_num 173) unflushed 28 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: finishing write tid 3 to smithi15938319-2 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: finishing write tid 4 to smithi15938319-2 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: finishing write tid 5 to smithi15938319-2 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: oid 2 updating version 0 to 208 2024-09-13T13:51:58.091 INFO:tasks.rados.rados.0.smithi159.stdout:1705: oid 2 updating version 208 to 209 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1705: oid 2 updating version 209 to 210 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1705: oid 2 updating version 210 to 211 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 2 v 211 (ObjNum 318 snap 46 seq_num 318) dirty exists 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1705: left oid 2 (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1707: finishing write tid 1 to smithi15938319-48 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1705: done (12 left) 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1706: done (11 left) 2024-09-13T13:51:58.092 INFO:tasks.rados.rados.0.smithi159.stdout:1719: read oid 26 snap -1 2024-09-13T13:51:58.093 INFO:tasks.rados.rados.0.smithi159.stdout:1719: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:51:58.093 INFO:tasks.rados.rados.0.smithi159.stdout:1720: set_chunk oid 9 target oid 2024-09-13T13:51:58.093 INFO:tasks.rados.rados.0.smithi159.stdout:1720: set_chunk oid 9 offset: 1991680 length: 66560 target oid cdb2033732975c26342b7b337f27c315c8fe1d87b1e0c34fbc56aae863e4fdfd offset: 0 snap -1 2024-09-13T13:51:58.093 INFO:tasks.rados.rados.0.smithi159.stdout:1721: write oid 40 current snap is 46 2024-09-13T13:51:58.094 INFO:tasks.rados.rados.0.smithi159.stdout:1707: finishing write tid 2 to smithi15938319-48 2024-09-13T13:51:58.094 INFO:tasks.rados.rados.0.smithi159.stdout:1707: finishing write tid 3 to smithi15938319-48 2024-09-13T13:51:58.094 INFO:tasks.rados.rados.0.smithi159.stdout:1721: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-13T13:51:58.099 INFO:tasks.rados.rados.0.smithi159.stdout:1721: writing smithi15938319-40 from 645468 to 1112970 tid 1 2024-09-13T13:51:58.102 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:58.102+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:58.105 INFO:tasks.rados.rados.0.smithi159.stdout:1721: writing smithi15938319-40 from 1726754 to 2292190 tid 2 2024-09-13T13:51:58.112 INFO:tasks.rados.rados.0.smithi159.stdout:1721: writing smithi15938319-40 from 2798226 to 3395064 tid 3 2024-09-13T13:51:58.112 INFO:tasks.rados.rados.0.smithi159.stdout:1722: read oid 45 snap -1 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1722: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1723: read oid 6 snap -1 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1723: expect (ObjNum 274 snap 37 seq_num 274) 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: finishing write tid 4 to smithi15938319-48 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: finishing write tid 5 to smithi15938319-48 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: oid 48 updating version 0 to 345 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: oid 48 updating version 345 to 346 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: oid 48 updating version 346 to 348 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: oid 48 version 348 is already newer than 347 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 348 (ObjNum 319 snap 46 seq_num 319) dirty exists 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1707: left oid 48 (ObjNum 319 snap 46 seq_num 319) 2024-09-13T13:51:58.113 INFO:tasks.rados.rados.0.smithi159.stdout:1708: finishing write tid 1 to smithi15938319-4 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: finishing write tid 2 to smithi15938319-4 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1707: done (15 left) 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1724: read oid 43 snap 40 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1724: expect (ObjNum 290 snap 38 seq_num 290) 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: finishing write tid 3 to smithi15938319-4 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: finishing write tid 4 to smithi15938319-4 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: finishing write tid 5 to smithi15938319-4 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: oid 4 updating version 0 to 293 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: oid 4 version 293 is already newer than 292 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: oid 4 version 293 is already newer than 291 2024-09-13T13:51:58.114 INFO:tasks.rados.rados.0.smithi159.stdout:1708: oid 4 version 293 is already newer than 290 2024-09-13T13:51:58.115 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 293 (ObjNum 320 snap 46 seq_num 320) dirty exists 2024-09-13T13:51:58.115 INFO:tasks.rados.rados.0.smithi159.stdout:1708: left oid 4 (ObjNum 320 snap 46 seq_num 320) 2024-09-13T13:51:58.115 INFO:tasks.rados.rados.0.smithi159.stdout:1709: finishing write tid 1 to smithi15938319-42 2024-09-13T13:51:58.115 INFO:tasks.rados.rados.0.smithi159.stdout:1708: done (15 left) 2024-09-13T13:51:58.115 INFO:tasks.rados.rados.0.smithi159.stdout:1725: set_chunk oid 1 target oid 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1725: set_chunk oid 1 offset: 1812480 length: 339456 target oid 0a2bd9b1560c44d2fa4dfcced22731132c44546c2c91cf5e82225fc40416894c offset: 0 snap -1 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: finishing write tid 2 to smithi15938319-42 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: finishing write tid 3 to smithi15938319-42 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: finishing write tid 4 to smithi15938319-42 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: finishing write tid 5 to smithi15938319-42 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: oid 42 updating version 0 to 161 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: oid 42 updating version 161 to 163 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: oid 42 version 163 is already newer than 162 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:1709: oid 42 updating version 163 to 164 2024-09-13T13:51:58.131 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 164 (ObjNum 321 snap 46 seq_num 321) dirty exists 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1709: left oid 42 (ObjNum 321 snap 46 seq_num 321) 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: finishing write tid 1 to smithi15938319-12 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1709: done (15 left) 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1726: read oid 33 snap -1 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1726: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: finishing write tid 2 to smithi15938319-12 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: finishing write tid 3 to smithi15938319-12 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: finishing write tid 4 to smithi15938319-12 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: finishing write tid 5 to smithi15938319-12 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: oid 12 updating version 0 to 243 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: oid 12 version 243 is already newer than 242 2024-09-13T13:51:58.132 INFO:tasks.rados.rados.0.smithi159.stdout:1710: oid 12 version 243 is already newer than 240 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1710: oid 12 version 243 is already newer than 241 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 243 (ObjNum 322 snap 46 seq_num 322) dirty exists 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1710: left oid 12 (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1712: finishing write tid 1 to smithi15938319-34 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1710: done (15 left) 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1727: write (excl) oid 25 current snap is 46 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1712: finishing write tid 2 to smithi15938319-34 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1712: finishing write tid 3 to smithi15938319-34 2024-09-13T13:51:58.133 INFO:tasks.rados.rados.0.smithi159.stdout:1727: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-13T13:51:58.136 INFO:tasks.rados.rados.0.smithi159.stdout:1727: writing smithi15938319-25 from 662275 to 1205026 tid 1 2024-09-13T13:51:58.142 INFO:tasks.rados.rados.0.smithi159.stdout:1727: writing smithi15938319-25 from 1785236 to 2210683 tid 2 2024-09-13T13:51:58.143 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.143 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1727: writing smithi15938319-25 from 2942002 to 3462275 tid 3 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: finishing write tid 4 to smithi15938319-34 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: finishing write tid 5 to smithi15938319-34 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: oid 34 updating version 0 to 229 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: oid 34 version 229 is already newer than 228 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: oid 34 updating version 229 to 235 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: oid 34 version 235 is already newer than 234 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 235 (ObjNum 323 snap 46 seq_num 323) dirty exists 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: left oid 34 (ObjNum 323 snap 46 seq_num 323) 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1713: finishing rollback tid 0 to smithi15938319-44 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1712: done (15 left) 2024-09-13T13:51:58.150 INFO:tasks.rados.rados.0.smithi159.stdout:1728: tier_evict oid 20 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1728: tier_evict oid 20 snap -1 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1713: finishing rollback tid 1 to smithi15938319-44 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1713: finishing rollback tid 2 to smithi15938319-44 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 247 (ObjNum 210 snap 22 seq_num 210) dirty exists 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1713: done (15 left) 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1729: write oid 44 current snap is 46 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1716: finishing rollback tid 1 to smithi15938319-15 2024-09-13T13:51:58.151 INFO:tasks.rados.rados.0.smithi159.stdout:1729: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-13T13:51:58.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:51:58.156 INFO:tasks.rados.rados.0.smithi159.stdout:1729: writing smithi15938319-44 from 679082 to 1297082 tid 1 2024-09-13T13:51:58.162 INFO:tasks.rados.rados.0.smithi159.stdout:1729: writing smithi15938319-44 from 1843718 to 2412823 tid 2 2024-09-13T13:51:58.167 INFO:tasks.rados.rados.0.smithi159.stdout:1729: writing smithi15938319-44 from 2853072 to 3479082 tid 3 2024-09-13T13:51:58.167 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 16 2024-09-13T13:51:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:1716: finishing rollback tid 2 to smithi15938319-15 2024-09-13T13:51:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 251 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:51:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:1715: finishing set_chunk to oid 29 2024-09-13T13:51:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 57 (ObjNum 87 snap 5 seq_num 87) dirty exists 2024-09-13T13:51:58.168 INFO:tasks.rados.rados.0.smithi159.stdout:1719: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:51:58.203 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:58.202+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:1718: got (0) Success 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 292 (ObjNum 292 snap 41 seq_num 292) dirty exists 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:1720: finishing set_chunk to oid 9 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 249 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:1714: expect (ObjNum 280 snap 38 seq_num 280) 2024-09-13T13:51:58.288 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.300 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.300 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.302 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:51:58.302 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:51:58.304 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:51:58.303+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:51:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:51:58.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:51:58.310 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:51:58.348 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 819 2024-09-13T13:51:58.378 INFO:tasks.rados.rados.0.smithi159.stdout:1721: finishing write tid 1 to smithi15938319-40 2024-09-13T13:51:58.378 INFO:tasks.rados.rados.0.smithi159.stdout:1717: expect (ObjNum 261 snap 30 seq_num 261) 2024-09-13T13:51:58.378 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.405 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:51:58.405+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:51:58.459 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.459 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:51:58.481 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1259030 2024-09-13T13:51:58.507 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:58.506+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1721: finishing write tid 2 to smithi15938319-40 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1721: finishing write tid 3 to smithi15938319-40 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1721: finishing write tid 4 to smithi15938319-40 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1714: done (15 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1715: done (14 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1716: done (13 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1717: done (12 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1718: done (11 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1719: done (10 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1720: done (9 left) 2024-09-13T13:51:58.510 INFO:tasks.rados.rados.0.smithi159.stdout:1730: tier_evict oid 24 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1730: tier_evict oid 24 snap -1 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: finishing write tid 5 to smithi15938319-40 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: oid 40 updating version 0 to 234 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: oid 40 version 234 is already newer than 231 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: oid 40 version 234 is already newer than 233 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: oid 40 version 234 is already newer than 232 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 234 (ObjNum 324 snap 46 seq_num 324) dirty exists 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1721: left oid 40 (ObjNum 324 snap 46 seq_num 324) 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:1723: expect (ObjNum 274 snap 37 seq_num 274) 2024-09-13T13:51:58.511 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.563 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.575 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.575 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:51:58.617 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.627 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:51:58.627+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:51:58.669 INFO:tasks.rados.rados.0.smithi159.stdout:1722: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:51:58.690 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.691 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:51:58.703 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.708 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:51:58.708+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:51:58.741 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:58.740+0000 7f2284f10640 -1 osd.10 136 handle_osd_map injecting map crc failure 2024-09-13T13:51:58.745 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.788 INFO:tasks.rados.rados.0.smithi159.stdout:1724: expect (ObjNum 290 snap 38 seq_num 290) 2024-09-13T13:51:58.788 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:58.809 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:58.809+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:58.846 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.847 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:51:58.880 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1972831 2024-09-13T13:51:58.911 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:51:58.909+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:51:58.961 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:58.962 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:58.970 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:51:58.993 INFO:tasks.rados.rados.0.smithi159.stdout:1726: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:51:58.996 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 713818 2024-09-13T13:51:59.011 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:59.010+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:59.016 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 510861 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:1725: finishing set_chunk to oid 1 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 148 (ObjNum 287 snap 38 seq_num 287) dirty exists 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:1727: finishing write tid 1 to smithi15938319-25 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:1727: finishing write tid 2 to smithi15938319-25 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:1721: done (9 left) 2024-09-13T13:51:59.067 INFO:tasks.rados.rados.0.smithi159.stdout:1722: done (8 left) 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1723: done (7 left) 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1724: done (6 left) 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1725: done (5 left) 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1726: done (4 left) 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1731: write (excl) oid 42 current snap is 46 2024-09-13T13:51:59.068 INFO:tasks.rados.rados.0.smithi159.stdout:1731: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-13T13:51:59.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:51:59.072 INFO:tasks.rados.rados.0.smithi159.stdout:1731: writing smithi15938319-42 from 695889 to 1105491 tid 1 2024-09-13T13:51:59.078 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.078 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.079 INFO:tasks.rados.rados.0.smithi159.stdout:1731: writing smithi15938319-42 from 1618553 to 2331316 tid 2 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1731: writing smithi15938319-42 from 2996848 to 3495889 tid 3 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1732: rollback oid 40 current snap is 46 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 40 to 37 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1728: got (22) Invalid argument 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: finishing write tid 3 to smithi15938319-25 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: finishing write tid 4 to smithi15938319-25 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: finishing write tid 5 to smithi15938319-25 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: oid 25 updating version 0 to 191 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: oid 25 updating version 191 to 192 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: oid 25 version 192 is already newer than 190 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:1727: oid 25 updating version 192 to 193 2024-09-13T13:51:59.087 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 193 (ObjNum 325 snap 46 seq_num 325) dirty exists 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1727: left oid 25 (ObjNum 325 snap 46 seq_num 325) 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: finishing write tid 1 to smithi15938319-44 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: finishing write tid 2 to smithi15938319-44 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: finishing write tid 3 to smithi15938319-44 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: finishing write tid 4 to smithi15938319-44 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: finishing write tid 5 to smithi15938319-44 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: oid 44 updating version 0 to 253 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: oid 44 updating version 253 to 254 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: oid 44 version 254 is already newer than 252 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: oid 44 updating version 254 to 255 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 255 (ObjNum 326 snap 46 seq_num 326) dirty exists 2024-09-13T13:51:59.088 INFO:tasks.rados.rados.0.smithi159.stdout:1729: left oid 44 (ObjNum 326 snap 46 seq_num 326) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1730: got (22) Invalid argument 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1727: done (5 left) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1728: done (4 left) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1729: done (3 left) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1730: done (2 left) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1733: read oid 32 snap -1 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1733: expect deleted 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1734: read oid 23 snap -1 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1734: expect (ObjNum 306 snap 43 seq_num 306) 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1735: tier_evict oid 30 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1735: tier_evict oid 30 snap -1 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1731: finishing write tid 1 to smithi15938319-42 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1731: finishing write tid 2 to smithi15938319-42 2024-09-13T13:51:59.089 INFO:tasks.rados.rados.0.smithi159.stdout:1736: read oid 11 snap 33 2024-09-13T13:51:59.090 INFO:tasks.rados.rados.0.smithi159.stdout:1736: expect (ObjNum 231 snap 25 seq_num 231) 2024-09-13T13:51:59.090 INFO:tasks.rados.rados.0.smithi159.stdout:1737: snap_create 2024-09-13T13:51:59.090 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:51:59.085+0000 7f26bcfe2640 -1 osd.3 136 handle_osd_map injecting map crc failure 2024-09-13T13:51:59.091 INFO:tasks.rados.rados.0.smithi159.stdout:1731: finishing write tid 3 to smithi15938319-42 2024-09-13T13:51:59.091 INFO:tasks.rados.rados.0.smithi159.stdout:1735: got (22) Invalid argument 2024-09-13T13:51:59.091 INFO:tasks.rados.rados.0.smithi159.stdout:1731: finishing write tid 4 to smithi15938319-42 2024-09-13T13:51:59.091 INFO:tasks.rados.rados.0.smithi159.stdout:1731: finishing write tid 5 to smithi15938319-42 2024-09-13T13:51:59.091 INFO:tasks.rados.rados.0.smithi159.stdout:1731: oid 42 updating version 0 to 172 2024-09-13T13:51:59.092 INFO:tasks.rados.rados.0.smithi159.stdout:1731: oid 42 version 172 is already newer than 171 2024-09-13T13:51:59.092 INFO:tasks.rados.rados.0.smithi159.stdout:1731: oid 42 updating version 172 to 173 2024-09-13T13:51:59.092 INFO:tasks.rados.rados.0.smithi159.stdout:1731: oid 42 updating version 173 to 174 2024-09-13T13:51:59.092 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 174 (ObjNum 327 snap 46 seq_num 327) dirty exists 2024-09-13T13:51:59.092 INFO:tasks.rados.rados.0.smithi159.stdout:1731: left oid 42 (ObjNum 327 snap 46 seq_num 327) 2024-09-13T13:51:59.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:51:59.096 INFO:tasks.rados.rados.0.smithi159.stdout:1734: expect (ObjNum 306 snap 43 seq_num 306) 2024-09-13T13:51:59.096 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 271713 2024-09-13T13:51:59.112 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:51:59.111+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:51:59.166 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1069009 2024-09-13T13:51:59.180 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:59.178+0000 7f4ffbd7a640 -1 osd.1 136 handle_osd_map injecting map crc failure 2024-09-13T13:51:59.207 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.207 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.213 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:59.212+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:59.215 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:51:59.264 INFO:tasks.rados.rados.0.smithi159.stdout:1732: finishing rollback tid 0 to smithi15938319-40 2024-09-13T13:51:59.264 INFO:tasks.rados.rados.0.smithi159.stdout:1732: finishing rollback tid 1 to smithi15938319-40 2024-09-13T13:51:59.264 INFO:tasks.rados.rados.0.smithi159.stdout:1732: finishing rollback tid 2 to smithi15938319-40 2024-09-13T13:51:59.264 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 237 (ObjNum 175 snap 16 seq_num 175) dirty exists 2024-09-13T13:51:59.265 INFO:tasks.rados.rados.0.smithi159.stdout:1736: expect (ObjNum 231 snap 25 seq_num 231) 2024-09-13T13:51:59.265 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:51:59.314 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:51:59.313+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:51:59.319 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.319 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:51:59.344 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1044248 2024-09-13T13:51:59.415 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:51:59.414+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:51:59.431 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.431 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:51:59.516 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:59.516+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:59.584 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.584 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:51:59.637 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:51:59.636+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:51:59.696 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.696 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.702 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:51:59.718 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:51:59.717+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:51:59.819 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:51:59.819+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:51:59.845 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.845 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.854 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:51:59.919 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:51:59.919+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:51:59.957 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:51:59.957 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:51:59.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:00.021 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:00.020+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:00.060 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1731: done (6 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1732: done (5 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1733: done (4 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1734: done (3 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1735: done (2 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1736: done (1 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1737: done (0 left) 2024-09-13T13:52:00.064 INFO:tasks.rados.rados.0.smithi159.stdout:1738: write (excl) oid 44 current snap is 47 2024-09-13T13:52:00.065 INFO:tasks.rados.rados.0.smithi159.stdout:1738: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-13T13:52:00.070 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:00.070 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:00.070 INFO:tasks.rados.rados.0.smithi159.stdout:1738: writing smithi15938319-44 from 712696 to 1197547 tid 1 2024-09-13T13:52:00.077 INFO:tasks.rados.rados.0.smithi159.stdout:1738: writing smithi15938319-44 from 1960682 to 2533456 tid 2 2024-09-13T13:52:00.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:00.082 INFO:tasks.rados.rados.0.smithi159.stdout:1738: writing smithi15938319-44 from 3307918 to 3512696 tid 3 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1739: set_chunk oid 5 target oid 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1739: set_chunk oid 5 offset: 0 length: 4000000 target oid smithi15938319- offset: 0 snap -1 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1740: read oid 18 snap -1 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1740: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1741: rollback oid 15 current snap is 47 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 15 to 34 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1738: finishing write tid 1 to smithi15938319-44 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1742: write (excl) oid 39 current snap is 47 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1738: finishing write tid 2 to smithi15938319-44 2024-09-13T13:52:00.083 INFO:tasks.rados.rados.0.smithi159.stdout:1742: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-13T13:52:00.089 INFO:tasks.rados.rados.0.smithi159.stdout:1742: writing smithi15938319-39 from 729503 to 1289603 tid 1 2024-09-13T13:52:00.097 INFO:tasks.rados.rados.0.smithi159.stdout:1742: writing smithi15938319-39 from 2019164 to 2735596 tid 2 2024-09-13T13:52:00.104 INFO:tasks.rados.rados.0.smithi159.stdout:1742: writing smithi15938319-39 from 3335341 to 3529503 tid 3 2024-09-13T13:52:00.104 INFO:tasks.rados.rados.0.smithi159.stdout:1743: set_chunk oid 45 target oid 2024-09-13T13:52:00.121 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:00.121+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:00.123 INFO:tasks.rados.rados.0.smithi159.stdout:1743: set_chunk oid 45 offset: 420352 length: 1573888 target oid 9f1a33d039d703dcfac2cb18f98c1dc8a72f99f8e74f9995df183dce14166d38 offset: 0 snap -1 2024-09-13T13:52:00.123 INFO:tasks.rados.rados.0.smithi159.stdout:1744: rollback oid 6 current snap is 47 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 6 to 35 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: finishing write tid 3 to smithi15938319-44 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1739: finishing set_chunk to oid 5 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1739: got expected ERANGE (src dne) 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: finishing write tid 4 to smithi15938319-44 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: finishing write tid 5 to smithi15938319-44 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: oid 44 updating version 0 to 260 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: oid 44 version 260 is already newer than 259 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: oid 44 version 260 is already newer than 258 2024-09-13T13:52:00.124 INFO:tasks.rados.rados.0.smithi159.stdout:1738: oid 44 version 260 is already newer than 257 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 260 (ObjNum 328 snap 47 seq_num 328) dirty exists 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1738: left oid 44 (ObjNum 328 snap 47 seq_num 328) 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1741: finishing rollback tid 0 to smithi15938319-15 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1738: done (6 left) 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1739: done (5 left) 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1745: tier_evict oid 44 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1745: tier_evict oid 44 snap -1 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1741: finishing rollback tid 1 to smithi15938319-15 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:1740: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:52:00.125 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:00.183 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:00.183 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:00.186 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:00.191 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:00.222 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:00.222+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:00.247 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 45698 2024-09-13T13:52:00.294 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:00.294 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:00.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:00.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:00.308 INFO:tasks.rados.rados.0.smithi159.stdout:1742: finishing write tid 1 to smithi15938319-39 2024-09-13T13:52:00.308 INFO:tasks.rados.rados.0.smithi159.stdout:1742: finishing write tid 2 to smithi15938319-39 2024-09-13T13:52:00.308 INFO:tasks.rados.rados.0.smithi159.stdout:1742: finishing write tid 3 to smithi15938319-39 2024-09-13T13:52:00.308 INFO:tasks.rados.rados.0.smithi159.stdout:1742: finishing write tid 4 to smithi15938319-39 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: finishing write tid 5 to smithi15938319-39 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: oid 39 updating version 0 to 242 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: oid 39 version 242 is already newer than 240 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: oid 39 version 242 is already newer than 239 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: oid 39 version 242 is already newer than 241 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 39 v 242 (ObjNum 329 snap 47 seq_num 329) dirty exists 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: left oid 39 (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1740: done (5 left) 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1742: done (4 left) 2024-09-13T13:52:00.309 INFO:tasks.rados.rados.0.smithi159.stdout:1746: snap_remove snap 28 2024-09-13T13:52:00.346 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:00.345+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:00.406 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:00.406 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:00.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:00.451 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:00.451 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:00.455 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:00.454+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:00.473 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:00.473 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:00.495 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:00.495 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:00.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:00.517 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:00.517 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:00.526 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:00.525+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:00.539 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:00.557 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:00.557 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:00.561 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:00.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:00.583 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:00.603 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:00.604 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:00.625 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:00.626 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:00.624+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:00.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:00.670 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:00.684 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:00.684 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:00.684 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:00.684 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:00.690 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:00.693 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:00.713 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:00.726 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:00.725+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:00.734 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:00.734 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:00.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:00.740+0000 7f7678d45640 0 -- 172.21.15.27:0/3637328288 >> v2:172.21.15.159:6805/4073091144 conn(0x7f76742223b0 msgr2=0x7f76742247a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:00.762 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:00.762 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:00.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:00.798 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:00.798 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:00.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:00.827 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:00.826+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:00.911 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.011004555, 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:52:00.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:52:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:52:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:52:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.99537694, 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:52:00.921 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:52:00.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.927 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.986665265, 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.928 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:52:00.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:52:00.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:00.934 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 92.162379114999993, 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:52:00.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:52:00.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 90.947708836000004, 2024-09-13T13:52:00.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 87.945907210000001, 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:52:00.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:52:00.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:00.949 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.666983642999995, 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:52:00.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:52:00.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.666960283999998, 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:00.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:52:00.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2156 4.1 4:8b8235b2:::smithi15938319-16:head [write 473997~627345,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 69.168515447000004, 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095295090000000002, 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2156 2024-09-13T13:52:00.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740655+0000", 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.740656+0000", 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741540+0000", 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088396500000000001 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741555+0000", 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4697e-05 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741562+0000", 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1099999999999997e-06 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741621+0000", 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9252999999999997e-05 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.741628+0000", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8689999999999998e-06 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743089+0000", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014604520000000001 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743158+0000", 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9659999999999994e-05 2024-09-13T13:52:00.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 12", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.743367+0000", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020889500000000001 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.749993+0000", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066259420000000001 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750062+0000", 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9145000000000003e-05 2024-09-13T13:52:00.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750127+0000", 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4567999999999996e-05 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:51.750184+0000", 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7228999999999997e-05 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.968 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.082894084000003, 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:52:00.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:52:00.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.973 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 50.14802384, 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:52:00.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:00.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2853 4.1 4:8e40678c:::smithi15938319-17:head [sparse-read 0~2202141,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.052139424000003, 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098372819999999993, 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.980 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2853 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857033+0000", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3149999999999999e-06 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857041+0000", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5369999999999997e-06 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857060+0000", 2024-09-13T13:52:00.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8972000000000001e-05 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857070+0000", 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3489999999999998e-06 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857145+0000", 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4870000000000007e-05 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857153+0000", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5310000000000001e-06 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.862925+0000", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057717510000000003 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.863022+0000", 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6618000000000002e-05 2024-09-13T13:52:00.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866868+0000", 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038463389999999998 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:00.985 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.052079525000003, 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:00.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:52:00.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:52:00.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 49.052041664000001, 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:52:00.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:52:00.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.071346792999996, 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:00.996 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:00.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:00.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:52:01.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.070954069000003, 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.003 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:52:01.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 48.070938468999998, 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:01.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3367 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e117)", 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.936169104000001, 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096522090000000001, 2024-09-13T13:52:01.015 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3367 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973001+0000", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973002+0000", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2799999999999995e-07 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973007+0000", 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9799999999999998e-06 2024-09-13T13:52:01.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973015+0000", 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973023+0000", 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1859999999999992e-06 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973140+0000", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011665100000000001 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.973146+0000", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4960000000000001e-06 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980078+0000", 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069321570000000004 2024-09-13T13:52:01.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.980132+0000", 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3834e-05 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:34.982653+0000", 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025209910000000002 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3399 4.1 4:80c58c2d:::smithi15938319-35:head [delete] snapc 27=[27,26,25,24,23,22,20,1e,1b,1a,17,14,12] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117)", 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.900588573, 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0088886150000000008, 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.021 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3399 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008581+0000", 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008582+0000", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4300000000000003e-07 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008584+0000", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3410000000000001e-06 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008591+0000", 2024-09-13T13:52:01.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4320000000000004e-06 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.008598+0000", 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015266+0000", 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066677849999999999 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015323+0000", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7108000000000003e-05 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.015457+0000", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013399700000000001 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017064+0000", 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016070769999999999 2024-09-13T13:52:01.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017414+0000", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00034987400000000002 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017445+0000", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0917999999999998e-05 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:35.017470+0000", 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5576e-05 2024-09-13T13:52:01.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3899 4.1 4:8bb695d2:::smithi15938319-48:head [write 561433~774904,stat] snapc 2f=[2f,2d,2c,2a,28,27,25,24,23,22,1e,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 3.6143910300000002, 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011332106, 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.027 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3899 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294781+0000", 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295899+0000", 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011175060000000001 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295916+0000", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7275e-05 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295925+0000", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2550000000000003e-06 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295992+0000", 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6973999999999999e-05 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295999+0000", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299822+0000", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038227019999999999 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299902+0000", 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0341999999999998e-05 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.300046+0000", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014437299999999999 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.304078+0000", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040313459999999999 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.305972+0000", 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018946200000000001 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306041+0000", 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8399999999999996e-05 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306111+0000", 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0492000000000002e-05 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.034 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:01.035 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:01.034+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:01.066 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:01.065+0000 7f2284f10640 -1 osd.10 138 handle_osd_map injecting map crc failure 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1746: done (4 left) 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1747: read oid 12 snap -1 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1747: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1748: read oid 14 snap 24 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1748: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1749: read oid 20 snap -1 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1749: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:52:01.071 INFO:tasks.rados.rados.0.smithi159.stdout:1750: set_chunk oid 9 target oid 2024-09-13T13:52:01.071 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:01.070+0000 7fe859f53640 -1 osd.7 138 handle_osd_map injecting map crc failure 2024-09-13T13:52:01.101 INFO:tasks.rados.rados.0.smithi159.stdout:1750: set_chunk oid 9 offset: 235008 length: 1997824 target oid 4c7b8c20deabd7648aef3ce82e07a21864ac97a9ea802bea6342259ea09b7b5e offset: 0 snap 35 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1751: tier_promote oid 22 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1745: got (22) Invalid argument 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1743: finishing set_chunk to oid 45 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 34 (ObjNum 295 snap 41 seq_num 295) dirty exists 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1744: finishing rollback tid 0 to smithi15938319-6 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1744: finishing rollback tid 1 to smithi15938319-6 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1744: finishing rollback tid 2 to smithi15938319-6 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 63 (ObjNum 253 snap 28 seq_num 253) dirty exists 2024-09-13T13:52:01.102 INFO:tasks.rados.rados.0.smithi159.stdout:1743: done (8 left) 2024-09-13T13:52:01.103 INFO:tasks.rados.rados.0.smithi159.stdout:1744: done (7 left) 2024-09-13T13:52:01.103 INFO:tasks.rados.rados.0.smithi159.stdout:1745: done (6 left) 2024-09-13T13:52:01.103 INFO:tasks.rados.rados.0.smithi159.stdout:1752: write oid 37 current snap is 47 2024-09-13T13:52:01.103 INFO:tasks.rados.rados.0.smithi159.stdout:1752: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-13T13:52:01.108 INFO:tasks.rados.rados.0.smithi159.stdout:1752: writing smithi15938319-37 from 746310 to 1381659 tid 1 2024-09-13T13:52:01.115 INFO:tasks.rados.rados.0.smithi159.stdout:1752: writing smithi15938319-37 from 2077646 to 2654089 tid 2 2024-09-13T13:52:01.119 INFO:tasks.rados.rados.0.smithi159.stdout:1752: writing smithi15938319-37 from 3362764 to 3546310 tid 3 2024-09-13T13:52:01.119 INFO:tasks.rados.rados.0.smithi159.stdout:1753: write oid 25 current snap is 47 2024-09-13T13:52:01.120 INFO:tasks.rados.rados.0.smithi159.stdout:1753: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-13T13:52:01.127 INFO:tasks.rados.rados.0.smithi159.stdout:1753: writing smithi15938319-25 from 763117 to 1473715 tid 1 2024-09-13T13:52:01.135 INFO:tasks.rados.rados.0.smithi159.stdout:1753: writing smithi15938319-25 from 2136128 to 2856229 tid 2 2024-09-13T13:52:01.135 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:01.134+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1753: writing smithi15938319-25 from 3390187 to 3563117 tid 3 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1754: read oid 40 snap 35 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1754: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1755: read oid 31 snap -1 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1755: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:52:01.139 INFO:tasks.rados.rados.0.smithi159.stdout:1756: read oid 7 snap -1 2024-09-13T13:52:01.140 INFO:tasks.rados.rados.0.smithi159.stdout:1756: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:52:01.140 INFO:tasks.rados.rados.0.smithi159.stdout:1757: write oid 49 current snap is 47 2024-09-13T13:52:01.140 INFO:tasks.rados.rados.0.smithi159.stdout:1741: finishing rollback tid 2 to smithi15938319-15 2024-09-13T13:52:01.140 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 264 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:52:01.140 INFO:tasks.rados.rados.0.smithi159.stdout:1747: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:01.179 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.180 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:01.180 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:01.180 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:01.213 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2185103 2024-09-13T13:52:01.226 INFO:tasks.rados.rados.0.smithi159.stdout:1749: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:52:01.226 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:01.236 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:01.234+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:01.291 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:01.292 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:01.336 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:01.335+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:01.393 INFO:tasks.rados.rados.0.smithi159.stdout:1748: expect (ObjNum 174 snap 16 seq_num 174) 2024-09-13T13:52:01.394 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:01.404 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 132.861293397, 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:52:01.405 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:01.406 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:01.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:52:01.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:52:01.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 130.80696750800001, 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:52:01.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:52:01.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:52:01.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:52:01.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 130.793589118, 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:52:01.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:52:01.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:01.423 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 102.980256181, 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:52:01.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:52:01.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174844174, 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.429 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:52:01.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:52:01.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174807001999994, 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:01.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:52:01.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174786273999999, 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.442 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:52:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:52:01.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:01.448 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174689973, 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174667329000002, 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.454 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:52:01.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:52:01.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.174641299000001, 2024-09-13T13:52:01.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:01.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.466 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.035140432000006, 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:01.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.473 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.035033485, 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.474 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:01.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:01.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:52:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:52:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:52:01.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:52:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:52:01.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:52:01.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:52:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:52:01.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.487 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.035004225999998, 2024-09-13T13:52:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:52:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.488 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.489 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:52:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:52:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:52:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:52:01.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:52:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:52:01.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:52:01.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:52:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:52:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.028639823000006, 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.502 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:52:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:52:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:52:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:52:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.023679629, 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:52:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:52:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:52:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:52:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 86.081689912000002, 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 86.081650773999996, 2024-09-13T13:52:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:52:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:52:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:52:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:52:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:52:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:52:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:52:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:52:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:52:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 59.540220777000002, 2024-09-13T13:52:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:52:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:52:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 59.540157831000002, 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:52:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:52:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:52:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 59.540124149, 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:52:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.559 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.561 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:01.559+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:01.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:01.580 INFO:tasks.rados.rados.0.smithi159.stdout:1751: got (0) Success 2024-09-13T13:52:01.580 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 22 v 49 (ObjNum 304 snap 43 seq_num 304) dirty exists 2024-09-13T13:52:01.580 INFO:tasks.rados.rados.0.smithi159.stdout:1750: finishing set_chunk to oid 9 2024-09-13T13:52:01.580 INFO:tasks.rados.rados.0.smithi159.stdout:1752: finishing write tid 1 to smithi15938319-37 2024-09-13T13:52:01.581 INFO:tasks.rados.rados.0.smithi159.stdout:1752: finishing write tid 2 to smithi15938319-37 2024-09-13T13:52:01.581 INFO:tasks.rados.rados.0.smithi159.stdout:1752: finishing write tid 3 to smithi15938319-37 2024-09-13T13:52:01.581 INFO:tasks.rados.rados.0.smithi159.stdout:1757: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-13T13:52:01.587 INFO:tasks.rados.rados.0.smithi159.stdout:1757: writing smithi15938319-49 from 779924 to 1565771 tid 1 2024-09-13T13:52:01.593 INFO:tasks.rados.rados.0.smithi159.stdout:1757: writing smithi15938319-49 from 2078257 to 2658369 tid 2 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1757: writing smithi15938319-49 from 3301257 to 3579924 tid 3 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1741: done (11 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1747: done (10 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1748: done (9 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1749: done (8 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1750: done (7 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1751: done (6 left) 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1758: tier_evict oid 17 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1758: tier_evict oid 17 snap -1 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1752: finishing write tid 4 to smithi15938319-37 2024-09-13T13:52:01.599 INFO:tasks.rados.rados.0.smithi159.stdout:1753: finishing write tid 1 to smithi15938319-25 2024-09-13T13:52:01.600 INFO:tasks.rados.rados.0.smithi159.stdout:1759: write (excl) oid 28 current snap is 47 2024-09-13T13:52:01.600 INFO:tasks.rados.rados.0.smithi159.stdout:1759: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-13T13:52:01.605 INFO:tasks.rados.rados.0.smithi159.stdout:1759: writing smithi15938319-28 from 796731 to 1257827 tid 1 2024-09-13T13:52:01.612 INFO:tasks.rados.rados.0.smithi159.stdout:1759: writing smithi15938319-28 from 1736739 to 2460509 tid 2 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1759: writing smithi15938319-28 from 2928680 to 3596731 tid 3 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1760: read oid 10 snap -1 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1760: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1761: read oid 34 snap -1 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1761: expect (ObjNum 323 snap 46 seq_num 323) 2024-09-13T13:52:01.618 INFO:tasks.rados.rados.0.smithi159.stdout:1762: write oid 19 current snap is 47 2024-09-13T13:52:01.619 INFO:tasks.rados.rados.0.smithi159.stdout:1762: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-13T13:52:01.623 INFO:tasks.rados.rados.0.smithi159.stdout:1762: writing smithi15938319-19 from 413538 to 949883 tid 1 2024-09-13T13:52:01.629 INFO:tasks.rados.rados.0.smithi159.stdout:1762: writing smithi15938319-19 from 1395221 to 1979002 tid 2 2024-09-13T13:52:01.636 INFO:tasks.rados.rados.0.smithi159.stdout:1762: writing smithi15938319-19 from 2556103 to 3214720 tid 3 2024-09-13T13:52:01.636 INFO:tasks.rados.rados.0.smithi159.stdout:1763: write (excl) oid 42 current snap is 47 2024-09-13T13:52:01.637 INFO:tasks.rados.rados.0.smithi159.stdout:1753: finishing write tid 2 to smithi15938319-25 2024-09-13T13:52:01.637 INFO:tasks.rados.rados.0.smithi159.stdout:1763: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-13T13:52:01.642 INFO:tasks.rados.rados.0.smithi159.stdout:1763: writing smithi15938319-42 from 430345 to 1158292 tid 1 2024-09-13T13:52:01.648 INFO:tasks.rados.rados.0.smithi159.stdout:1763: writing smithi15938319-42 from 1570056 to 2297495 tid 2 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1763: writing smithi15938319-42 from 2699879 to 3169768 tid 3 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1764: read oid 35 snap -1 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1764: expect deleted 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1765: write oid 5 current snap is 47 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1753: finishing write tid 3 to smithi15938319-25 2024-09-13T13:52:01.653 INFO:tasks.rados.rados.0.smithi159.stdout:1765: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-13T13:52:01.657 INFO:tasks.rados.rados.0.smithi159.stdout:1765: writing smithi15938319-5 from 447152 to 850348 tid 1 2024-09-13T13:52:01.661 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:01.660+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:01.664 INFO:tasks.rados.rados.0.smithi159.stdout:1765: writing smithi15938319-5 from 1512185 to 2099635 tid 2 2024-09-13T13:52:01.672 INFO:tasks.rados.rados.0.smithi159.stdout:1765: writing smithi15938319-5 from 2727302 to 3408463 tid 3 2024-09-13T13:52:01.672 INFO:tasks.rados.rados.0.smithi159.stdout:1766: tier_evict oid 21 2024-09-13T13:52:01.672 INFO:tasks.rados.rados.0.smithi159.stdout:1766: tier_evict oid 21 snap -1 2024-09-13T13:52:01.672 INFO:tasks.rados.rados.0.smithi159.stdout:1753: finishing write tid 4 to smithi15938319-25 2024-09-13T13:52:01.672 INFO:tasks.rados.rados.0.smithi159.stdout:1753: finishing write tid 5 to smithi15938319-25 2024-09-13T13:52:01.673 INFO:tasks.rados.rados.0.smithi159.stdout:1753: oid 25 updating version 0 to 195 2024-09-13T13:52:01.673 INFO:tasks.rados.rados.0.smithi159.stdout:1753: oid 25 updating version 195 to 202 2024-09-13T13:52:01.673 INFO:tasks.rados.rados.0.smithi159.stdout:1753: oid 25 version 202 is already newer than 200 2024-09-13T13:52:01.673 INFO:tasks.rados.rados.0.smithi159.stdout:1753: oid 25 version 202 is already newer than 201 2024-09-13T13:52:01.673 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 202 (ObjNum 331 snap 47 seq_num 331) dirty exists 2024-09-13T13:52:01.674 INFO:tasks.rados.rados.0.smithi159.stdout:1753: left oid 25 (ObjNum 331 snap 47 seq_num 331) 2024-09-13T13:52:01.674 INFO:tasks.rados.rados.0.smithi159.stdout:1754: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:01.676 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.676 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:01.676 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:01.676 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.685 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:01.757 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:01.761 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:01.760+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:01.787 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.787 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:01.787 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:01.787 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:01.787 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:01.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:01.810 INFO:tasks.rados.rados.0.smithi159.stdout:1756: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:52:01.860 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2547419 2024-09-13T13:52:01.862 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:01.861+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:01.896 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1124829 2024-09-13T13:52:01.900 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.16166824, 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.901 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:52:01.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:52:01.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.156427618, 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.907 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:52:01.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:52:01.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:52:01.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.15637630400001, 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.913 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:01.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.918 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.156304107, 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:52:01.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:52:01.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.15626510200001, 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:52:01.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:52:01.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:52:01.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.15620069000001, 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.931 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:52:01.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:52:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.13085059399999, 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.937 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:52:01.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:52:01.942 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.10028977, 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.943 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:52:01.948 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.078963183, 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.949 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:52:01.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.000746906, 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:01.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:52:01.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:52:01.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:01.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.959 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656775245999995, 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:52:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:52:01.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656717193999995, 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:01.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:52:01.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656691234999997, 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:52:01.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656576818999994, 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.978 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:52:01.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:52:01.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656545928, 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.985 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:52:01.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:52:01.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:52:01.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:52:01.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.656509345000003, 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:01.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:52:01.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:01.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.998 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.576714475000003, 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:01.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:52:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.202054056999998, 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.003 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:52:02.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.078862534000002, 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:52:02.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:52:02.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:02.012 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.070842235999997, 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:02.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.017 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.017 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:02.018 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:02.017+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:02.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:02.040 INFO:tasks.rados.rados.0.smithi159.stdout:1755: expect (ObjNum 138 snap 11 seq_num 138) 2024-09-13T13:52:02.055 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1269715 2024-09-13T13:52:02.075 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:02.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:52:02.118 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:02.118 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:02.117+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:02.122 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:02.122 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:02.123 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:02.123 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:02.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:1752: finishing write tid 5 to smithi15938319-37 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:1752: oid 37 updating version 0 to 221 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:1752: oid 37 updating version 221 to 224 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:1752: oid 37 version 224 is already newer than 223 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:1752: oid 37 version 224 is already newer than 222 2024-09-13T13:52:02.161 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 224 (ObjNum 330 snap 47 seq_num 330) dirty exists 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1752: left oid 37 (ObjNum 330 snap 47 seq_num 330) 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: finishing write tid 1 to smithi15938319-49 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: finishing write tid 2 to smithi15938319-49 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1758: got (22) Invalid argument 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: finishing write tid 3 to smithi15938319-49 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: finishing write tid 4 to smithi15938319-49 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: finishing write tid 5 to smithi15938319-49 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: oid 49 updating version 0 to 76 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: oid 49 version 76 is already newer than 75 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: oid 49 version 76 is already newer than 74 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: oid 49 version 76 is already newer than 73 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 76 (ObjNum 332 snap 47 seq_num 332) dirty exists 2024-09-13T13:52:02.162 INFO:tasks.rados.rados.0.smithi159.stdout:1757: left oid 49 (ObjNum 332 snap 47 seq_num 332) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1759: finishing write tid 1 to smithi15938319-28 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1752: done (14 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1753: done (13 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1754: done (12 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1755: done (11 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1756: done (10 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1757: done (9 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1758: done (8 left) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1767: read oid 14 snap -1 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1767: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1768: tier_evict oid 40 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1768: tier_evict oid 40 snap -1 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1759: finishing write tid 2 to smithi15938319-28 2024-09-13T13:52:02.163 INFO:tasks.rados.rados.0.smithi159.stdout:1759: finishing write tid 3 to smithi15938319-28 2024-09-13T13:52:02.164 INFO:tasks.rados.rados.0.smithi159.stdout:1769: set_chunk oid 6 target oid 2024-09-13T13:52:02.170 INFO:tasks.rados.rados.0.smithi159.stdout:1769: set_chunk oid 6 offset: 939520 length: 935424 target oid 094bb230396427062f282ae85750f3fc235a8de285e2882255e637a57a15f2b4 offset: 0 snap 32 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1770: snap_remove snap 45 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: finishing write tid 4 to smithi15938319-28 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: finishing write tid 5 to smithi15938319-28 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: oid 28 updating version 0 to 228 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: oid 28 version 228 is already newer than 227 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: oid 28 version 228 is already newer than 226 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: oid 28 version 228 is already newer than 225 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 28 v 228 (ObjNum 333 snap 47 seq_num 333) dirty exists 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1759: left oid 28 (ObjNum 333 snap 47 seq_num 333) 2024-09-13T13:52:02.171 INFO:tasks.rados.rados.0.smithi159.stdout:1760: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:02.216 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:02.219 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:02.219+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:02.234 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:02.235 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:02.235 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:02.235 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:02.235 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:02.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:02.290 INFO:tasks.rados.rados.0.smithi159.stdout:1761: expect (ObjNum 323 snap 46 seq_num 323) 2024-09-13T13:52:02.320 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:02.319+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:02.349 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:02.349 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:02.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:02.349 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 132.117830653, 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.350 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:52:02.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:52:02.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:52:02.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.613092471, 2024-09-13T13:52:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.359 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:52:02.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:52:02.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:52:02.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:52:02.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:52:02.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:52:02.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:52:02.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.368 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.613043159, 2024-09-13T13:52:02.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:02.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:52:02.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:52:02.375 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.61302745499999, 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.376 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:52:02.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:52:02.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:52:02.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.383 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.61299977900001, 2024-09-13T13:52:02.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.385 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:52:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:52:02.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.577782181, 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:52:02.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:52:02.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.570713199, 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:52:02.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:52:02.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.536174737, 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.407 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:52:02.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:52:02.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.413 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.528544895, 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.414 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:52:02.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:52:02.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.525168523, 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.420 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:52:02.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:02.426 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.422309243, 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:02.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:52:02.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:52:02.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:02.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.432 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.211285703, 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:02.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:52:02.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.19977041999999, 2024-09-13T13:52:02.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:02.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:52:02.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.098461951999994, 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.442 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:02.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.447 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.098397603999999, 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:52:02.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:52:02.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:52:02.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.098369542, 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:52:02.454 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:02.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:02.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.096687652, 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.460 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:52:02.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:52:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:52:02.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:52:02.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.468 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.657159444000001, 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:52:02.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.472 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.528523688999996, 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:52:02.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.478 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 31.383267089, 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:02.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:02.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:52:02.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:52:02.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:52:02.486 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:02.487 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:02.487 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:02.488 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 390826 2024-09-13T13:52:02.488 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:02.487+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:02.489 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: finishing write tid 1 to smithi15938319-19 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: finishing write tid 2 to smithi15938319-19 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: finishing write tid 3 to smithi15938319-19 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: finishing write tid 4 to smithi15938319-19 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: finishing write tid 5 to smithi15938319-19 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: oid 19 updating version 0 to 207 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: oid 19 updating version 207 to 208 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: oid 19 updating version 208 to 209 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:1762: oid 19 version 209 is already newer than 206 2024-09-13T13:52:02.530 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 209 (ObjNum 334 snap 47 seq_num 334) dirty exists 2024-09-13T13:52:02.531 INFO:tasks.rados.rados.0.smithi159.stdout:1762: left oid 19 (ObjNum 334 snap 47 seq_num 334) 2024-09-13T13:52:02.531 INFO:tasks.rados.rados.0.smithi159.stdout:1763: finishing write tid 1 to smithi15938319-42 2024-09-13T13:52:02.531 INFO:tasks.rados.rados.0.smithi159.stdout:1763: finishing write tid 2 to smithi15938319-42 2024-09-13T13:52:02.589 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:02.588+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:02.593 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:02.593 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:02.593 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:02.594 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:02.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:02.689 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:02.689+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:02.747 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:02.747 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:02.747 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:02.747 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:02.747 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:02.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:02.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:02.790 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:02.789+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:02.891 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:02.891+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:02.900 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:02.900 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.16650269199999, 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.901 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:02.902 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.903 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.904 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:52:02.905 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.906 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.166425291, 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.907 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:02.908 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.909 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.910 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.911 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:52:02.912 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.166272753, 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.913 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.914 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:52:02.915 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:52:02.916 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.917 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.918 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.166166714, 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:52:02.919 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.920 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:52:02.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:52:02.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.15782763599999, 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.925 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.926 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:52:02.927 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:52:02.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.130460926, 2024-09-13T13:52:02.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:52:02.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:52:02.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.12869432799999, 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:52:02.935 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:52:02.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:52:02.938 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.123814986, 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:52:02.941 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:52:02.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.947 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.113303261, 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.948 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:52:02.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.089045971, 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:52:02.954 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.956 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:52:02.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.961 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.962 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.08616092600001, 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.963 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:52:02.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:52:02.968 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.969 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.08395690099999, 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.970 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.971 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.971 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:52:02.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.971 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:02.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:52:02.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:52:02.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.076966257, 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:52:02.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:52:02.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:02.981 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 109.971974643, 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:52:02.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:52:02.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.148384781999994, 2024-09-13T13:52:02.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:02.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:52:02.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:52:02.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.148351992000002, 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:52:02.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:02.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:52:02.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:02.999 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.032592484999995, 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:03.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:52:03.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:52:03.003 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 67.107989795999998, 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:03.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:03.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:03.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:52:03.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:03.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.010 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.157780049000003, 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:03.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:52:03.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:03.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3286 3.2 e114/73 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head v 114'133, mlcod=105'131)", 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 31.930810757, 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74551292800000002, 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:03.015 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:03.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.73e-07 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.968632+0000", 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075455699999999999 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711912+0000", 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74328065200000004 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711933+0000", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1279000000000001e-05 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712080+0000", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014671199999999999 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712101+0000", 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0339999999999998e-05 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713357+0000", 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012564550000000001 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713390+0000", 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2759999999999998e-05 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:03.019 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:03.020 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:03.021 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:03.021 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:03.022 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:03.022 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:03.023 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:03.023 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:03.020+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:03.023 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:03.024 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:03.024 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:03.025 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:03.025 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:03.025 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:03.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:03.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:03.075 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:52:03.079 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:03.079 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:03.079 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:03.079+0000 7faeaa210640 -1 osd.8 139 handle_osd_map injecting map crc failure 2024-09-13T13:52:03.080 INFO:tasks.rados.rados.0.smithi159.stdout:1759: done (11 left) 2024-09-13T13:52:03.080 INFO:tasks.rados.rados.0.smithi159.stdout:1760: done (10 left) 2024-09-13T13:52:03.080 INFO:tasks.rados.rados.0.smithi159.stdout:1761: done (9 left) 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1762: done (8 left) 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1770: done (7 left) 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1771: read oid 22 snap -1 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1771: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1772: read oid 9 snap -1 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1772: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1773: write (excl) oid 41 current snap is 47 2024-09-13T13:52:03.081 INFO:tasks.rados.rados.0.smithi159.stdout:1773: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-13T13:52:03.085 INFO:tasks.rados.rados.0.smithi159.stdout:1773: writing smithi15938319-41 from 463959 to 942404 tid 1 2024-09-13T13:52:03.093 INFO:tasks.rados.rados.0.smithi159.stdout:1773: writing smithi15938319-41 from 1570667 to 2301775 tid 2 2024-09-13T13:52:03.100 INFO:tasks.rados.rados.0.smithi159.stdout:1773: writing smithi15938319-41 from 3038372 to 3530805 tid 3 2024-09-13T13:52:03.100 INFO:tasks.rados.rados.0.smithi159.stdout:1774: read oid 17 snap 46 2024-09-13T13:52:03.100 INFO:tasks.rados.rados.0.smithi159.stdout:1774: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:52:03.100 INFO:tasks.rados.rados.0.smithi159.stdout:1775: snap_remove snap 37 2024-09-13T13:52:03.105 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:03.105 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:03.122 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:03.121+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:03.127 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:03.147 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:03.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:03.168 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.169 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.169 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:03.170 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:03.177 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:03.191 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:03.191 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:03.191+0000 7f65d2ffd640 0 -- 172.21.15.27:0/161707340 >> v2:172.21.15.159:6805/4073091144 conn(0x7f65d4220170 msgr2=0x7f65d4222560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:03.263 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:03.262+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:03.273 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:03.272+0000 7f7c71c1d640 -1 osd.14 139 handle_osd_map injecting map crc failure 2024-09-13T13:52:03.321 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.321 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.323 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:03.323+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:03.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:03.414 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:03.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:03.425 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:03.424+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:03.474 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.474 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.483 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:03.526 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:03.525+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:03.625 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.625 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.626 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:03.625+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:03.633 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:03.727 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:03.726+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:03.737 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.737 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1775: done (11 left) 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1776: write (excl) oid 1 current snap is 47 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1763: finishing write tid 3 to smithi15938319-42 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1763: finishing write tid 4 to smithi15938319-42 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1763: finishing write tid 5 to smithi15938319-42 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1763: oid 42 updating version 0 to 183 2024-09-13T13:52:03.747 INFO:tasks.rados.rados.0.smithi159.stdout:1763: oid 42 updating version 183 to 185 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1763: oid 42 version 185 is already newer than 182 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1763: oid 42 version 185 is already newer than 184 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 185 (ObjNum 335 snap 47 seq_num 335) dirty exists 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1763: left oid 42 (ObjNum 335 snap 47 seq_num 335) 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1765: finishing write tid 1 to smithi15938319-5 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1765: finishing write tid 2 to smithi15938319-5 2024-09-13T13:52:03.748 INFO:tasks.rados.rados.0.smithi159.stdout:1776: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-13T13:52:03.750 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:03.748+0000 7fe0f9e03640 -1 osd.2 140 handle_osd_map injecting map crc failure 2024-09-13T13:52:03.751 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:03.750+0000 7f2282f0c640 -1 osd.10 140 handle_osd_map injecting map crc failure 2024-09-13T13:52:03.752 INFO:tasks.rados.rados.0.smithi159.stdout:1776: writing smithi15938319-1 from 480766 to 1034460 tid 1 2024-09-13T13:52:03.758 INFO:tasks.rados.rados.0.smithi159.stdout:1776: writing smithi15938319-1 from 1629149 to 2220268 tid 2 2024-09-13T13:52:03.760 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:03.759+0000 7fc16a180640 -1 osd.11 139 handle_osd_map injecting map crc failure 2024-09-13T13:52:03.764 INFO:tasks.rados.rados.0.smithi159.stdout:1776: writing smithi15938319-1 from 2782148 to 3202206 tid 3 2024-09-13T13:52:03.764 INFO:tasks.rados.rados.0.smithi159.stdout:1763: done (11 left) 2024-09-13T13:52:03.764 INFO:tasks.rados.rados.0.smithi159.stdout:1764: done (10 left) 2024-09-13T13:52:03.765 INFO:tasks.rados.rados.0.smithi159.stdout:1777: read oid 15 snap -1 2024-09-13T13:52:03.765 INFO:tasks.rados.rados.0.smithi159.stdout:1777: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:52:03.765 INFO:tasks.rados.rados.0.smithi159.stdout:1778: read oid 4 snap -1 2024-09-13T13:52:03.765 INFO:tasks.rados.rados.0.smithi159.stdout:1778: expect (ObjNum 320 snap 46 seq_num 320) 2024-09-13T13:52:03.765 INFO:tasks.rados.rados.0.smithi159.stdout:1779: set_chunk oid 26 target oid 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1779: set_chunk oid 26 offset: 431616 length: 1174016 target oid 1007c59501ad872140d09381db60c29f9f138d075816fb79358a36cea413192f offset: 0 snap 40 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1780: write (excl) oid 24 current snap is 47 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1766: got (22) Invalid argument 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1765: finishing write tid 3 to smithi15938319-5 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1765: finishing write tid 4 to smithi15938319-5 2024-09-13T13:52:03.783 INFO:tasks.rados.rados.0.smithi159.stdout:1768: got (22) Invalid argument 2024-09-13T13:52:03.784 INFO:tasks.rados.rados.0.smithi159.stdout:1780: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-13T13:52:03.790 INFO:tasks.rados.rados.0.smithi159.stdout:1780: writing smithi15938319-24 from 497573 to 1126516 tid 1 2024-09-13T13:52:03.798 INFO:tasks.rados.rados.0.smithi159.stdout:1780: writing smithi15938319-24 from 1687631 to 2422408 tid 2 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1780: writing smithi15938319-24 from 3093218 to 3697573 tid 3 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1766: done (13 left) 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1768: done (12 left) 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1781: read oid 11 snap -1 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1781: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1782: read oid 2 snap 38 2024-09-13T13:52:03.807 INFO:tasks.rados.rados.0.smithi159.stdout:1782: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:52:03.808 INFO:tasks.rados.rados.0.smithi159.stdout:1783: delete oid 36 current snap is 47 2024-09-13T13:52:03.808 INFO:tasks.rados.rados.0.smithi159.stdout:1767: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:03.828 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:03.828+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:03.864 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.864 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.873 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:03.890 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1463958 2024-09-13T13:52:03.935 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:03.934+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:03.995 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:03.995 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:03.998 INFO:tasks.rados.rados.0.smithi159.stdout:1769: finishing set_chunk to oid 6 2024-09-13T13:52:03.998 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 939520 length: 935424 2024-09-13T13:52:03.998 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 0 (ObjNum 253 snap 28 seq_num 253) dirty exists 2024-09-13T13:52:03.998 INFO:tasks.rados.rados.0.smithi159.stdout:1765: finishing write tid 5 to smithi15938319-5 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1765: oid 5 updating version 0 to 246 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1765: oid 5 version 246 is already newer than 244 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1765: oid 5 version 246 is already newer than 245 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1765: oid 5 updating version 246 to 247 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 247 (ObjNum 336 snap 47 seq_num 336) dirty exists 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1765: left oid 5 (ObjNum 336 snap 47 seq_num 336) 2024-09-13T13:52:03.999 INFO:tasks.rados.rados.0.smithi159.stdout:1771: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:52:04.004 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:04.045 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:04.044+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:04.131 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:04.130+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:04.148 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.148 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.157 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:04.188 INFO:tasks.rados.rados.0.smithi159.stdout:1772: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:04.200 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1309374 2024-09-13T13:52:04.232 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:04.231+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:04.263 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.263 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:04.283 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:04.333 INFO:tasks.rados.rados.0.smithi159.stdout:1773: finishing write tid 1 to smithi15938319-41 2024-09-13T13:52:04.333 INFO:tasks.rados.rados.0.smithi159.stdout:1773: finishing write tid 2 to smithi15938319-41 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: finishing write tid 3 to smithi15938319-41 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: finishing write tid 4 to smithi15938319-41 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: finishing write tid 5 to smithi15938319-41 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: oid 41 updating version 0 to 274 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: oid 41 updating version 274 to 278 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: oid 41 updating version 278 to 279 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: oid 41 version 279 is already newer than 277 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 279 (ObjNum 337 snap 47 seq_num 337) dirty exists 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1773: left oid 41 (ObjNum 337 snap 47 seq_num 337) 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1765: done (14 left) 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1767: done (13 left) 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1769: done (12 left) 2024-09-13T13:52:04.334 INFO:tasks.rados.rados.0.smithi159.stdout:1771: done (11 left) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1772: done (10 left) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1773: done (9 left) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1783: done (8 left) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1784: read oid 50 snap 38 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1784: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1785: tier_evict oid 36 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1785: tier_evict oid 36 snap -1 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:1774: expect (ObjNum 292 snap 41 seq_num 292) 2024-09-13T13:52:04.335 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:04.336 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:04.332+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:04.390 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.390 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:04.399 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:04.434 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:04.432+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:04.507 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.507 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: finishing write tid 1 to smithi15938319-1 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: finishing write tid 2 to smithi15938319-1 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: finishing write tid 3 to smithi15938319-1 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: finishing write tid 4 to smithi15938319-1 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: finishing write tid 5 to smithi15938319-1 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: oid 1 updating version 0 to 224 2024-09-13T13:52:04.528 INFO:tasks.rados.rados.0.smithi159.stdout:1776: oid 1 version 224 is already newer than 223 2024-09-13T13:52:04.529 INFO:tasks.rados.rados.0.smithi159.stdout:1776: oid 1 version 224 is already newer than 215 2024-09-13T13:52:04.529 INFO:tasks.rados.rados.0.smithi159.stdout:1776: oid 1 version 224 is already newer than 218 2024-09-13T13:52:04.529 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 224 (ObjNum 338 snap 47 seq_num 338) dirty exists 2024-09-13T13:52:04.529 INFO:tasks.rados.rados.0.smithi159.stdout:1776: left oid 1 (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:04.529 INFO:tasks.rados.rados.0.smithi159.stdout:1778: expect (ObjNum 320 snap 46 seq_num 320) 2024-09-13T13:52:04.534 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:04.533+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:04.623 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2366423 2024-09-13T13:52:04.623 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.623 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:04.635 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:04.633+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:04.682 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2564534 2024-09-13T13:52:04.701 INFO:tasks.rados.rados.0.smithi159.stdout:1777: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:52:04.735 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:04.735+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:04.739 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.739 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:04.780 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2046710 2024-09-13T13:52:04.792 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:04.837 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:04.836+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:04.844 INFO:tasks.rados.rados.0.smithi159.stdout:1779: finishing set_chunk to oid 26 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 431616 length: 1174016 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 26 v 0 (ObjNum 311 snap 45 seq_num 311) dirty exists 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: finishing write tid 1 to smithi15938319-24 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: finishing write tid 2 to smithi15938319-24 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: finishing write tid 3 to smithi15938319-24 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: finishing write tid 4 to smithi15938319-24 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: finishing write tid 5 to smithi15938319-24 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: oid 24 updating version 0 to 275 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: oid 24 updating version 275 to 276 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: oid 24 updating version 276 to 278 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:1780: oid 24 version 278 is already newer than 277 2024-09-13T13:52:04.845 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 278 (ObjNum 339 snap 47 seq_num 339) dirty exists 2024-09-13T13:52:04.846 INFO:tasks.rados.rados.0.smithi159.stdout:1780: left oid 24 (ObjNum 339 snap 47 seq_num 339) 2024-09-13T13:52:04.846 INFO:tasks.rados.rados.0.smithi159.stdout:1781: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:04.846 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:04.854 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.854 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:04.938 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:04.936+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:04.970 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:04.970 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:04.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:05.001 INFO:tasks.rados.rados.0.smithi159.stdout:1782: expect (ObjNum 237 snap 26 seq_num 237) 2024-09-13T13:52:05.039 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:05.037+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:05.087 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:05.088 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:05.091 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:52:05.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:05.140 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:05.139+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:05.197 INFO:tasks.rados.rados.0.smithi159.stdout:1785: got (22) Invalid argument 2024-09-13T13:52:05.197 INFO:tasks.rados.rados.0.smithi159.stdout:1784: expect (ObjNum 268 snap 33 seq_num 268) 2024-09-13T13:52:05.241 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:05.240+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:05.246 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:05.254 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:05.254 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:05.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:05.273 INFO:tasks.rados.rados.0.smithi159.stdout:1774: done (9 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1776: done (8 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1777: done (7 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1778: done (6 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1779: done (5 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1780: done (4 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1781: done (3 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1782: done (2 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1784: done (1 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1785: done (0 left) 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1786: rollback oid 46 current snap is 47 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 46 to 33 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1787: write (excl) oid 15 current snap is 47 2024-09-13T13:52:05.274 INFO:tasks.rados.rados.0.smithi159.stdout:1787: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-13T13:52:05.280 INFO:tasks.rados.rados.0.smithi159.stdout:1787: writing smithi15938319-15 from 531187 to 1310628 tid 1 2024-09-13T13:52:05.288 INFO:tasks.rados.rados.0.smithi159.stdout:1787: writing smithi15938319-15 from 2088242 to 2826688 tid 2 2024-09-13T13:52:05.293 INFO:tasks.rados.rados.0.smithi159.stdout:1787: writing smithi15938319-15 from 3431711 to 3731187 tid 3 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1788: read oid 12 snap 46 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1788: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1789: read oid 49 snap 33 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1789: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1790: write (excl) oid 36 current snap is 47 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1786: finishing rollback tid 0 to smithi15938319-46 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1786: finishing rollback tid 1 to smithi15938319-46 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1786: finishing rollback tid 2 to smithi15938319-46 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 148 (ObjNum 247 snap 28 seq_num 247) dirty exists 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1787: finishing write tid 1 to smithi15938319-15 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1787: finishing write tid 2 to smithi15938319-15 2024-09-13T13:52:05.294 INFO:tasks.rados.rados.0.smithi159.stdout:1790: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-13T13:52:05.298 INFO:tasks.rados.rados.0.smithi159.stdout:1790: writing smithi15938319-36 from 547994 to 1002684 tid 1 2024-09-13T13:52:05.305 INFO:tasks.rados.rados.0.smithi159.stdout:1790: writing smithi15938319-36 from 1746724 to 2228828 tid 2 2024-09-13T13:52:05.310 INFO:tasks.rados.rados.0.smithi159.stdout:1790: writing smithi15938319-36 from 2659134 to 3124280 tid 3 2024-09-13T13:52:05.314 INFO:tasks.rados.rados.0.smithi159.stdout:1790: writing smithi15938319-36 from 3605976 to 3747994 tid 4 2024-09-13T13:52:05.314 INFO:tasks.rados.rados.0.smithi159.stdout:1786: done (4 left) 2024-09-13T13:52:05.314 INFO:tasks.rados.rados.0.smithi159.stdout:1791: write (excl) oid 23 current snap is 47 2024-09-13T13:52:05.314 INFO:tasks.rados.rados.0.smithi159.stdout:1791: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-13T13:52:05.320 INFO:tasks.rados.rados.0.smithi159.stdout:1791: writing smithi15938319-23 from 564801 to 1211093 tid 1 2024-09-13T13:52:05.327 INFO:tasks.rados.rados.0.smithi159.stdout:1791: writing smithi15938319-23 from 1921559 to 2663674 tid 2 2024-09-13T13:52:05.333 INFO:tasks.rados.rados.0.smithi159.stdout:1791: writing smithi15938319-23 from 3202910 to 3764801 tid 3 2024-09-13T13:52:05.333 INFO:tasks.rados.rados.0.smithi159.stdout:1792: snap_remove snap 24 2024-09-13T13:52:05.342 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:05.341+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:05.371 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:05.371 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:05.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:05.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:05.463 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:05.462+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:05.497 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:05.497 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:05.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:05.543 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:05.542+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:05.552 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:05.552 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:05.574 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:05.574 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:05.597 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:05.597 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:05.609 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:05.609 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:05.609 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:05.609 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:05.609 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:05.618 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:05.619 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:05.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:05.619 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:05.640 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:05.640 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:05.645 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:05.644+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:05.662 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:05.662 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:05.683 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:05.684 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:05.704 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:05.723 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:05.723 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 112.806552806, 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:05.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:52:05.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 112.801753277, 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:52:05.730 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:52:05.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:52:05.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:05.736 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 111.952405239, 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:52:05.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:52:05.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:350 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 111.64760492800001, 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008982973, 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.743 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 350 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074239+0000", 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074240+0000", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.15e-06 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074245+0000", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2089999999999996e-06 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074256+0000", 2024-09-13T13:52:05.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0971e-05 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074262+0000", 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1279999999999996e-06 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074304+0000", 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1545000000000003e-05 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.074356+0000", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2528999999999998e-05 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079418+0000", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050623780000000002 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.079855+0000", 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000436997 2024-09-13T13:52:05.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083128+0000", 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003272559 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083178+0000", 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0411000000000002e-05 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.083221+0000", 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3096000000000002e-05 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.448107639, 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:52:05.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:52:05.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.442247690000002, 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.757 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:52:05.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:52:05.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:52:05.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.437303048999993, 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:52:05.765 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:05.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:52:05.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:52:05.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:52:05.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.437092471, 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.773 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:52:05.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:05.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:52:05.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.780 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.437044372000003, 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.781 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:05.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:52:05.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.353963961000005, 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:52:05.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.353855249999995, 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.794 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:52:05.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.353739970000007, 2024-09-13T13:52:05.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:52:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:52:05.804 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.479768867999994, 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:05.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:05.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.479682550999996, 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:05.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.479622180999996, 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.815 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:52:05.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:05.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 63.845770035999998, 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:52:05.820 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:52:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:52:05.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:52:05.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3008 4.7 4:e26a40dd:::smithi15938319-25:head [sparse-read 0~3142009] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.518575589000001, 2024-09-13T13:52:05.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0093824860000000006, 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3008 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203268+0000", 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203272+0000", 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6770000000000001e-06 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203280+0000", 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.203287+0000", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0149999999999998e-06 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210677+0000", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073896709999999996 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.210743+0000", 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6392999999999999e-05 2024-09-13T13:52:05.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:18.212650+0000", 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019070300000000001 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3134 4.7 4:e1bd516b:::smithi15938319-19:head [write 769820~720742,stat] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 43.548929432000001, 2024-09-13T13:52:05.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096203899999999995, 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3134 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:05.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172917+0000", 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6759999999999999e-06 2024-09-13T13:52:05.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173729+0000", 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081220499999999998 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173743+0000", 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4073000000000001e-05 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173749+0000", 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173849+0000", 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9809999999999995e-05 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173917+0000", 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7556000000000002e-05 2024-09-13T13:52:05.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.174067+0000", 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000150335 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.180026+0000", 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059590329999999999 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182455+0000", 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024288000000000001 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182498+0000", 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2861000000000002e-05 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182534+0000", 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6522e-05 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.838 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(osd.1.0:130 4.7 e130/52 ondisk, result = 0)", 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.907552597, 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18760375600000001, 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814292+0000", 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7999999999999995e-07 2024-09-13T13:52:05.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814296+0000", 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8339999999999998e-06 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001621+0000", 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18732465400000001 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001634+0000", 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2883e-05 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001774+0000", 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000140104 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001805+0000", 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001895+0000", 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9456000000000005e-05 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4040 4.7 4:e26a40dd:::smithi15938319-25:head [write 763117~710598] snapc 30=[30,2f,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5938247499999996, 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010818598, 2024-09-13T13:52:05.843 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4040 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128020+0000", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.778e-06 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128710+0000", 2024-09-13T13:52:05.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000689637 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128727+0000", 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6628000000000001e-05 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128734+0000", 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128785+0000", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0828999999999997e-05 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128789+0000", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1409999999999998e-06 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133000+0000", 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042116080000000004 2024-09-13T13:52:05.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133067+0000", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6644999999999996e-05 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133188+0000", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121249 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138729+0000", 2024-09-13T13:52:05.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055408610000000002 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138749+0000", 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0029000000000001e-05 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138796+0000", 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6681999999999998e-05 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:05.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138837+0000", 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1440000000000003e-05 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:05.850 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:05.851 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:05.852 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:05.850+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:05.852 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:05.853 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:05.853 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:05.853 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:05.854 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:05.746+0000 7f586a715640 -1 osd.9 141 handle_osd_map injecting map crc failure 2024-09-13T13:52:05.854 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:05.854 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:05.855 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:05.855 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:05.855 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:05.856 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:05.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:05.835+0000 7ff930cbf640 0 -- 172.21.15.27:0/770238173 >> v2:172.21.15.159:6805/4073091144 conn(0x7ff92c2223f0 msgr2=0x7ff92c224800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:05.858 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:05.859 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:05.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:05.953 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:05.952+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:05.959 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:05.960 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:05.960 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:05.960 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:05.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:06.054 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:06.053+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:06.071 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:06.071 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:06.071 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:06.083 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:52:06.086 INFO:tasks.rados.rados.0.smithi159.stdout:1792: done (5 left) 2024-09-13T13:52:06.087 INFO:tasks.rados.rados.0.smithi159.stdout:1793: read oid 29 snap -1 2024-09-13T13:52:06.087 INFO:tasks.rados.rados.0.smithi159.stdout:1793: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:06.087 INFO:tasks.rados.rados.0.smithi159.stdout:1794: write oid 44 current snap is 47 2024-09-13T13:52:06.087 INFO:tasks.rados.rados.0.smithi159.stdout:1794: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-13T13:52:06.093 INFO:tasks.rados.rados.0.smithi159.stdout:1794: writing smithi15938319-44 from 581608 to 1303149 tid 1 2024-09-13T13:52:06.099 INFO:tasks.rados.rados.0.smithi159.stdout:1794: writing smithi15938319-44 from 1863688 to 2349461 tid 2 2024-09-13T13:52:06.102 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:06.100+0000 7f1a506ea640 -1 osd.13 142 handle_osd_map injecting map crc failure 2024-09-13T13:52:06.105 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:06.104+0000 7f2282f0c640 -1 osd.10 142 handle_osd_map injecting map crc failure 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1794: writing smithi15938319-44 from 3113980 to 3718023 tid 3 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1795: tier_evict oid 1 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1795: tier_evict oid 1 snap -1 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: finishing write tid 3 to smithi15938319-15 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: finishing write tid 4 to smithi15938319-15 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: finishing write tid 5 to smithi15938319-15 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: oid 15 updating version 0 to 286 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: oid 15 updating version 286 to 289 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: oid 15 version 289 is already newer than 288 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:1787: oid 15 version 289 is already newer than 287 2024-09-13T13:52:06.108 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 289 (ObjNum 341 snap 47 seq_num 341) dirty exists 2024-09-13T13:52:06.109 INFO:tasks.rados.rados.0.smithi159.stdout:1787: left oid 15 (ObjNum 341 snap 47 seq_num 341) 2024-09-13T13:52:06.109 INFO:tasks.rados.rados.0.smithi159.stdout:1788: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:06.125 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3376624 2024-09-13T13:52:06.155 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:06.154+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:06.163 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2805736 2024-09-13T13:52:06.174 INFO:tasks.rados.rados.0.smithi159.stdout:1789: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.186 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 112.415475875, 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.187 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:52:06.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.192 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 112.406445443, 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:52:06.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 107.75443956700001, 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.197 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.201 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 107.751405581, 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:52:06.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:52:06.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 90.875573462999995, 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:52:06.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.209 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 90.869666631000001, 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:06.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:52:06.214 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 90.862231195999996, 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:06.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.942937454000003, 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.219 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:06.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:52:06.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:52:06.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.942868789000002, 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:52:06.225 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:52:06.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:52:06.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.942818674999998, 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:52:06.231 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:52:06.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:52:06.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:06.237 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.921861313999997, 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:52:06.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.243 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.916265111000001, 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:52:06.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:52:06.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.249 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.914721201000006, 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:52:06.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:52:06.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:52:06.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:06.257 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.914656152999996, 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:52:06.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.262 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 84.307714554, 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.263 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 75.452038850999998, 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:52:06.272 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 75.380143163, 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.273 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:52:06.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:52:06.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 75.380105642000004, 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:06.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:52:06.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:52:06.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:52:06.284 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 74.267186886999994, 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:52:06.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:52:06.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 70.053120479, 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:52:06.291 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:52:06.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:52:06.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:52:06.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.297 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.299 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 1 to smithi15938319-36 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 2 to smithi15938319-36 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1787: done (7 left) 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1788: done (6 left) 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1789: done (5 left) 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1796: read oid 4 snap -1 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1796: expect (ObjNum 320 snap 46 seq_num 320) 2024-09-13T13:52:06.301 INFO:tasks.rados.rados.0.smithi159.stdout:1797: tier_promote oid 21 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 3 to smithi15938319-36 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 4 to smithi15938319-36 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 5 to smithi15938319-36 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: finishing write tid 6 to smithi15938319-36 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: oid 36 updating version 0 to 258 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: oid 36 updating version 258 to 261 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: oid 36 version 261 is already newer than 260 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: oid 36 updating version 261 to 262 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: oid 36 version 262 is already newer than 259 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 262 (ObjNum 342 snap 47 seq_num 342) dirty exists 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: left oid 36 (ObjNum 342 snap 47 seq_num 342) 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1791: finishing write tid 1 to smithi15938319-23 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1790: done (6 left) 2024-09-13T13:52:06.302 INFO:tasks.rados.rados.0.smithi159.stdout:1798: write oid 46 current snap is 47 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: finishing write tid 2 to smithi15938319-23 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: finishing write tid 3 to smithi15938319-23 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: finishing write tid 4 to smithi15938319-23 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: finishing write tid 5 to smithi15938319-23 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: oid 23 updating version 0 to 267 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: oid 23 version 267 is already newer than 265 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: oid 23 version 267 is already newer than 264 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: oid 23 version 267 is already newer than 266 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 267 (ObjNum 343 snap 47 seq_num 343) dirty exists 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1791: left oid 23 (ObjNum 343 snap 47 seq_num 343) 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1794: finishing write tid 1 to smithi15938319-44 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:1793: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:06.303 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:06.304 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:06.297+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:06.399 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:06.398+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:06.442 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.442 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:06.442 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:06.442 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.445 INFO:tasks.rados.rados.0.smithi159.stdout:1794: finishing write tid 2 to smithi15938319-44 2024-09-13T13:52:06.445 INFO:tasks.rados.rados.0.smithi159.stdout:1795: got (22) Invalid argument 2024-09-13T13:52:06.445 INFO:tasks.rados.rados.0.smithi159.stdout:1794: finishing write tid 3 to smithi15938319-44 2024-09-13T13:52:06.445 INFO:tasks.rados.rados.0.smithi159.stdout:1798: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-13T13:52:06.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:06.452 INFO:tasks.rados.rados.0.smithi159.stdout:1798: writing smithi15938319-46 from 598415 to 1395205 tid 1 2024-09-13T13:52:06.459 INFO:tasks.rados.rados.0.smithi159.stdout:1798: writing smithi15938319-46 from 1922170 to 2667954 tid 2 2024-09-13T13:52:06.464 INFO:tasks.rados.rados.0.smithi159.stdout:1798: writing smithi15938319-46 from 3141403 to 3556718 tid 3 2024-09-13T13:52:06.464 INFO:tasks.rados.rados.0.smithi159.stdout:1791: done (6 left) 2024-09-13T13:52:06.464 INFO:tasks.rados.rados.0.smithi159.stdout:1793: done (5 left) 2024-09-13T13:52:06.464 INFO:tasks.rados.rados.0.smithi159.stdout:1795: done (4 left) 2024-09-13T13:52:06.464 INFO:tasks.rados.rados.0.smithi159.stdout:1799: set_chunk oid 48 target oid 2024-09-13T13:52:06.472 INFO:tasks.rados.rados.0.smithi159.stdout:1799: set_chunk oid 48 offset: 2240512 length: 606208 target oid 3bd374cb96082a4e676df6e2c5f10afc476ef704177c5bee0f7b9ee9bf4eca10 offset: 0 snap -1 2024-09-13T13:52:06.472 INFO:tasks.rados.rados.0.smithi159.stdout:1800: rollback oid 16 current snap is 47 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 16 to 43 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: finishing write tid 4 to smithi15938319-44 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: finishing write tid 5 to smithi15938319-44 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: oid 44 updating version 0 to 301 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: oid 44 version 301 is already newer than 300 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: oid 44 version 301 is already newer than 295 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: oid 44 version 301 is already newer than 290 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 301 (ObjNum 344 snap 47 seq_num 344) dirty exists 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: left oid 44 (ObjNum 344 snap 47 seq_num 344) 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1794: done (5 left) 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1801: snap_create 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:1797: got (0) Success 2024-09-13T13:52:06.473 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 147 (ObjNum 305 snap 43 seq_num 305) dirty exists 2024-09-13T13:52:06.474 INFO:tasks.rados.rados.0.smithi159.stdout:1796: expect (ObjNum 320 snap 46 seq_num 320) 2024-09-13T13:52:06.500 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:06.498+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:06.511 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2800382 2024-09-13T13:52:06.530 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:06.553 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:06.553 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:06.553 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.562 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:06.600 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:06.600+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: finishing write tid 1 to smithi15938319-46 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: finishing write tid 2 to smithi15938319-46 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: finishing write tid 3 to smithi15938319-46 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: finishing write tid 4 to smithi15938319-46 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: finishing write tid 5 to smithi15938319-46 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: oid 46 updating version 0 to 151 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: oid 46 updating version 151 to 154 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: oid 46 version 154 is already newer than 152 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:1798: oid 46 version 154 is already newer than 153 2024-09-13T13:52:06.668 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 154 (ObjNum 345 snap 47 seq_num 345) dirty exists 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:1798: left oid 46 (ObjNum 345 snap 47 seq_num 345) 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:1800: finishing rollback tid 1 to smithi15938319-16 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:1800: finishing rollback tid 2 to smithi15938319-16 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 389 (ObjNum 199 snap 18 seq_num 199) dirty exists 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:1799: finishing set_chunk to oid 48 2024-09-13T13:52:06.669 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 348 (ObjNum 319 snap 46 seq_num 319) dirty exists 2024-09-13T13:52:06.701 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:06.700+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:52:06.705 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 135.47048294000001, 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:52:06.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 134.46244879400001, 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:52:06.710 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:52:06.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.066750635000005, 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:52:06.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.718 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.519846932999997, 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:52:06.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.723 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.515006108999998, 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:52:06.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:52:06.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.514293133999999, 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:06.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:52:06.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:06.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:06.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:06.732 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.733 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:06.802 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:06.800+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:06.836 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.837 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:06.837 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:06.837 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:06.903 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:06.902+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:06.988 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:06.989 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:06.989 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:06.989 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:06.989 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:06.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:07.004 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:07.004+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:07.089 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:07.088+0000 7f8f5ac65640 -1 osd.6 143 handle_osd_map injecting map crc failure 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1796: done (5 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1797: done (4 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1798: done (3 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1799: done (2 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1800: done (1 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1801: done (0 left) 2024-09-13T13:52:07.092 INFO:tasks.rados.rados.0.smithi159.stdout:1802: read oid 38 snap -1 2024-09-13T13:52:07.093 INFO:tasks.rados.rados.0.smithi159.stdout:1802: expect (ObjNum 303 snap 43 seq_num 303) 2024-09-13T13:52:07.093 INFO:tasks.rados.rados.0.smithi159.stdout:1803: write oid 32 current snap is 48 2024-09-13T13:52:07.093 INFO:tasks.rados.rados.0.smithi159.stdout:1803: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-13T13:52:07.097 INFO:tasks.rados.rados.0.smithi159.stdout:1803: writing smithi15938319-32 from 615222 to 1087261 tid 1 2024-09-13T13:52:07.102 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:07.102 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:07.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:07.102 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:07.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.333667809, 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:52:07.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:52:07.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:52:07.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.109 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.332690611, 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:07.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:52:07.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:52:07.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.330935946, 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:07.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:52:07.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.330543733, 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.122 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:07.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:52:07.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.330506406, 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:52:07.128 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:52:07.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:52:07.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.134 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.325708053, 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:52:07.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.310072828, 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.141 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:52:07.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:52:07.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:52:07.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.30996647800001, 2024-09-13T13:52:07.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:07.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:52:07.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.302032708, 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:07.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:52:07.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:52:07.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.28364415, 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.160 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:52:07.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:52:07.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.28351736, 2024-09-13T13:52:07.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:52:07.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:52:07.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.202318937, 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:52:07.172 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:52:07.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.202217486, 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.179 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:52:07.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:52:07.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:52:07.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 102.64086472, 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.185 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:52:07.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:52:07.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:52:07.191 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 102.635013177, 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:07.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1290 4.f 4:f883ee18:::smithi15938319-40:head [tier-promote] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 98.358203540999995, 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016207833000000001, 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:07.196 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1290 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742268+0000", 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742269+0000", 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5990000000000001e-06 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742275+0000", 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742290+0000", 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4786999999999999e-05 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742297+0000", 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1910000000000003e-06 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742353+0000", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5726000000000001e-05 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.742441+0000", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8141999999999997e-05 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.758476+0000", 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016034274000000001 2024-09-13T13:52:07.200 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 93.373135726000001, 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:07.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:52:07.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:07.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.207 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 92.367114451000006, 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:52:07.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.7.23:20 3.1 3:8dbd923e:::ae8b86d07d375a083683440b6bda0e0f67b5195d585764d7ac407424ac114e59:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:52:07.213 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 25.083679278000002, 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89571672999999996, 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016793+0000", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6600000000000005e-07 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016800+0000", 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1199999999999996e-06 2024-09-13T13:52:07.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910876+0000", 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89407579400000003 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910884+0000", 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910982+0000", 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7763999999999994e-05 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911101+0000", 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119942 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911338+0000", 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023690000000000001 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912020+0000", 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068117299999999998 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912443+0000", 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042320999999999998 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:07.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912478+0000", 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4768999999999999e-05 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912509+0000", 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1226999999999998e-05 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.219 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(3.1 126/125 [PushReplyOp(3:ac2af9f5:::e6b4f9f8265bfe5a786cb2bd77c53a79b3e484041c7c0186a813c80a2460efe6:head)])", 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.054260466999999, 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208631544, 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:07.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9399999999999995e-07 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046216+0000", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3959999999999999e-06 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254500+0000", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208283991 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254505+0000", 2024-09-13T13:52:07.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.789e-06 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254594+0000", 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8101000000000002e-05 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254842+0000", 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024877299999999998 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.224 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:07.224 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:07.224 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:07.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1803: writing smithi15938319-32 from 1580652 to 2070094 tid 2 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1803: writing smithi15938319-32 from 2768826 to 3395413 tid 3 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1804: read oid 33 snap -1 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1804: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1805: tier_evict oid 40 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1805: tier_evict oid 40 snap -1 2024-09-13T13:52:07.227 INFO:tasks.rados.rados.0.smithi159.stdout:1803: finishing write tid 1 to smithi15938319-32 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1802: expect (ObjNum 303 snap 43 seq_num 303) 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 818652 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: finishing write tid 2 to smithi15938319-32 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1805: got (22) Invalid argument 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: finishing write tid 3 to smithi15938319-32 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: finishing write tid 4 to smithi15938319-32 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: finishing write tid 5 to smithi15938319-32 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: oid 32 updating version 0 to 171 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: oid 32 version 171 is already newer than 170 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: oid 32 version 171 is already newer than 168 2024-09-13T13:52:07.228 INFO:tasks.rados.rados.0.smithi159.stdout:1803: oid 32 version 171 is already newer than 169 2024-09-13T13:52:07.229 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 171 (ObjNum 346 snap 48 seq_num 346) dirty exists 2024-09-13T13:52:07.229 INFO:tasks.rados.rados.0.smithi159.stdout:1803: left oid 32 (ObjNum 346 snap 48 seq_num 346) 2024-09-13T13:52:07.229 INFO:tasks.rados.rados.0.smithi159.stdout:1804: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:52:07.229 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1775901 2024-09-13T13:52:07.229 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:07.224+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:07.257 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1802: done (3 left) 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1803: done (2 left) 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1804: done (1 left) 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1805: done (0 left) 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1806: rollback oid 15 current snap is 48 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 15 to 34 2024-09-13T13:52:07.287 INFO:tasks.rados.rados.0.smithi159.stdout:1807: set_chunk oid 11 target oid 2024-09-13T13:52:07.290 INFO:tasks.rados.rados.0.smithi159.stdout:1807: set_chunk oid 11 offset: 2278912 length: 10752 target oid 6c1be3fa213c4d5df75767765b6afc851c06d212062bf6b7d2646545f032784d offset: 0 snap -1 2024-09-13T13:52:07.290 INFO:tasks.rados.rados.0.smithi159.stdout:1808: write oid 30 current snap is 48 2024-09-13T13:52:07.290 INFO:tasks.rados.rados.0.smithi159.stdout:1808: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-13T13:52:07.296 INFO:tasks.rados.rados.0.smithi159.stdout:1808: writing smithi15938319-30 from 632029 to 1179317 tid 1 2024-09-13T13:52:07.303 INFO:tasks.rados.rados.0.smithi159.stdout:1808: writing smithi15938319-30 from 1639134 to 2388587 tid 2 2024-09-13T13:52:07.307 INFO:tasks.rados.rados.0.smithi159.stdout:1808: writing smithi15938319-30 from 2796249 to 3234108 tid 3 2024-09-13T13:52:07.310 INFO:tasks.rados.rados.0.smithi159.stdout:1808: writing smithi15938319-30 from 3653524 to 3832029 tid 4 2024-09-13T13:52:07.310 INFO:tasks.rados.rados.0.smithi159.stdout:1809: set_chunk oid 31 target oid 2024-09-13T13:52:07.325 INFO:tasks.rados.rados.0.smithi159.stdout:1809: set_chunk oid 31 offset: 781824 length: 1328128 target oid acf7ec644eca0e63d81035c1acebcdd7613063452682e473aad3655f7a8f2e2f offset: 0 snap -1 2024-09-13T13:52:07.325 INFO:tasks.rados.rados.0.smithi159.stdout:1810: read oid 38 snap 38 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1810: expect deleted 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1811: tier_flush oid 39 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1811: tier_flush oid 39 snap -1 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1806: finishing rollback tid 0 to smithi15938319-15 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1806: finishing rollback tid 1 to smithi15938319-15 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1806: finishing rollback tid 2 to smithi15938319-15 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 305 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1806: done (5 left) 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1812: read oid 25 snap 33 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1812: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1813: read oid 46 snap -1 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1813: expect (ObjNum 345 snap 47 seq_num 345) 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1814: read oid 37 snap -1 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1814: expect (ObjNum 330 snap 47 seq_num 330) 2024-09-13T13:52:07.326 INFO:tasks.rados.rados.0.smithi159.stdout:1815: tier_evict oid 13 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1815: tier_evict oid 13 snap -1 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1807: finishing set_chunk to oid 11 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 192 (ObjNum 272 snap 36 seq_num 272) dirty exists 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 1 to smithi15938319-30 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1807: done (8 left) 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1816: read oid 29 snap 38 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1816: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:07.327 INFO:tasks.rados.rados.0.smithi159.stdout:1817: set_chunk oid 36 target oid 2024-09-13T13:52:07.328 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:07.327+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:07.333 INFO:tasks.rados.rados.0.smithi159.stdout:1817: set_chunk oid 36 offset: 3243520 length: 145920 target oid b840d32315a14a89e9e2dd7ae721b4f37181be64c48890a86501f9c087937823 offset: 0 snap -1 2024-09-13T13:52:07.333 INFO:tasks.rados.rados.0.smithi159.stdout:1818: snap_create 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 2 to smithi15938319-30 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 3 to smithi15938319-30 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 4 to smithi15938319-30 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 5 to smithi15938319-30 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: finishing write tid 6 to smithi15938319-30 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: oid 30 updating version 0 to 319 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: oid 30 updating version 319 to 321 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: oid 30 version 321 is already newer than 318 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: oid 30 version 321 is already newer than 320 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: oid 30 updating version 321 to 322 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 322 (ObjNum 347 snap 48 seq_num 347) dirty exists 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1808: left oid 30 (ObjNum 347 snap 48 seq_num 347) 2024-09-13T13:52:07.334 INFO:tasks.rados.rados.0.smithi159.stdout:1809: finishing set_chunk to oid 31 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 781824 length: 1328128 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:1815: got (22) Invalid argument 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:1811: got (16) Device or resource busy 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 39 v 242 (ObjNum 329 snap 47 seq_num 329) unflushed 47 2024-09-13T13:52:07.335 INFO:tasks.rados.rados.0.smithi159.stdout:1813: expect (ObjNum 345 snap 47 seq_num 345) 2024-09-13T13:52:07.360 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:07.360 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:07.360 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:07.360 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:07.366 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3512116 2024-09-13T13:52:07.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:07.429 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:07.428+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:07.472 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:07.472 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:07.472 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:07.473 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:07.473 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:07.480 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:07.487 INFO:tasks.rados.rados.0.smithi159.stdout:1812: expect (ObjNum 97 snap 6 seq_num 97) 2024-09-13T13:52:07.530 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:07.528+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:07.562 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.584 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 114.683861855, 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.585 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 114.66823424, 2024-09-13T13:52:07.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:52:07.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:52:07.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:52:07.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:07.599 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 114.659522565, 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.600 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:52:07.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:52:07.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.606 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 98.835236414999997, 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:52:07.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:52:07.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:52:07.612 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 97.620566135999994, 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.613 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:52:07.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:52:07.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:07.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 94.618764510000005, 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.618 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:52:07.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 86.339840942999999, 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.623 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:07.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:52:07.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 86.339817584000002, 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.629 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:07.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:52:07.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 66.755751384000007, 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:52:07.634 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:52:07.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:52:07.638 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.820881139999997, 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.639 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:52:07.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2853 4.1 4:8e40678c:::smithi15938319-17:head [sparse-read 0~2202141,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:07.645 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.724996724, 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098372819999999993, 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2853 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:07.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857031+0000", 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857033+0000", 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3149999999999999e-06 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857041+0000", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5369999999999997e-06 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857060+0000", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8972000000000001e-05 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857070+0000", 2024-09-13T13:52:07.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3489999999999998e-06 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857145+0000", 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4870000000000007e-05 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857153+0000", 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5310000000000001e-06 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.862925+0000", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057717510000000003 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.863022+0000", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6618000000000002e-05 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866868+0000", 2024-09-13T13:52:07.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038463389999999998 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.724936825, 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.651 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:52:07.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:07.656 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 55.724898963999998, 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:52:07.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:52:07.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.744204093, 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.662 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:52:07.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:07.668 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.743811368999999, 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:52:07.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:52:07.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.674 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.743795769000002, 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:52:07.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:52:07.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3899 4.1 4:8bb695d2:::smithi15938319-48:head [write 561433~774904,stat] snapc 2f=[2f,2d,2c,2a,28,27,25,24,23,22,1e,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 10.287248330000001, 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011332106, 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:07.683 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3899 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294781+0000", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295899+0000", 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011175060000000001 2024-09-13T13:52:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295916+0000", 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7275e-05 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295925+0000", 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2550000000000003e-06 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295992+0000", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6973999999999999e-05 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295999+0000", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299822+0000", 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038227019999999999 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299902+0000", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0341999999999998e-05 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.300046+0000", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014437299999999999 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.304078+0000", 2024-09-13T13:52:07.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040313459999999999 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.305972+0000", 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018946200000000001 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:07.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306041+0000", 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8399999999999996e-05 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306111+0000", 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0492000000000002e-05 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.690 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4169 4.1 4:81ab2613:::smithi15938319-29:head [read 0~3462209,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.4950796550000001, 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012063419000000001, 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4169 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086949+0000", 2024-09-13T13:52:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086957+0000", 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6350000000000006e-06 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086972+0000", 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5503e-05 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086982+0000", 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0009e-05 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087052+0000", 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9832999999999994e-05 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087058+0000", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091635+0000", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045774259999999999 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091765+0000", 2024-09-13T13:52:07.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012954199999999999 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099011+0000", 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072461840000000001 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4170 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:07.696 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.4950218470000001, 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014864408000000001, 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4170 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087006+0000", 2024-09-13T13:52:07.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0100000000000005e-07 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087008+0000", 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4200000000000001e-06 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087015+0000", 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087018+0000", 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7359999999999999e-06 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087126+0000", 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107847 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087134+0000", 2024-09-13T13:52:07.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2849999999999995e-06 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099058+0000", 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011923375 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099212+0000", 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015423100000000001 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101870+0000", 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002657899 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4171 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.4949862229999999, 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016120193000000001, 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:07.702 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4171 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:07.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3700000000000001e-07 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087044+0000", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087048+0000", 2024-09-13T13:52:07.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3170000000000003e-06 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087053+0000", 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.609e-06 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087165+0000", 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111686 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087169+0000", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6600000000000001e-06 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101916+0000", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014746795 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.102043+0000", 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012732600000000001 2024-09-13T13:52:07.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.103161+0000", 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011183759999999999 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:07.707 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:07.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:07.709 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:07.708+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:07.743 INFO:tasks.rados.rados.0.smithi159.stdout:1814: expect (ObjNum 330 snap 47 seq_num 330) 2024-09-13T13:52:07.743 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.797 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.810 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:07.809+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:07.852 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:07.852 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:07.853 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:07.853 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:07.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:07.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:07.895 INFO:tasks.rados.rados.0.smithi159.stdout:1817: finishing set_chunk to oid 36 2024-09-13T13:52:07.895 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 262 (ObjNum 342 snap 47 seq_num 342) dirty exists 2024-09-13T13:52:07.895 INFO:tasks.rados.rados.0.smithi159.stdout:1816: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:07.895 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:07.910 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:07.910+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:07.963 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:07.963 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:07.963 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:07.963 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:07.964 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:07.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:07.973 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3023364 2024-09-13T13:52:07.979 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 640310 2024-09-13T13:52:08.011 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:08.011 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:08.011 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:08.010+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:08.034 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:08.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:08.056 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:08.077 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 139.533764867, 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:52:08.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:08.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 137.47943897799999, 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.084 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:08.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:52:08.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:52:08.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.090 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 137.466060588, 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:52:08.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:52:08.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.094 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 109.65272765100001, 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.095 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:52:08.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.099 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847315644000005, 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.100 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:52:08.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.105 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847278471999999, 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.106 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:52:08.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:52:08.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:08.111 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847257744000004, 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:52:08.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:52:08.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847161443000005, 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.117 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:52:08.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847138799000007, 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.123 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:52:08.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:52:08.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:52:08.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:08.129 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.847112769000006, 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:08.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:52:08.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.707611901999996, 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:52:08.135 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:08.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:52:08.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:52:08.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.141 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.707504955000005, 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:08.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:52:08.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:52:08.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.707475696000003, 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:52:08.147 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:08.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:08.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.153 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.701111292999997, 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:52:08.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:08.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.696151099000005, 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.162 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:08.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:52:08.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:52:08.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 92.754161382000007, 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:08.170 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:08.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:52:08.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 92.754122244000001, 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:08.176 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:08.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:52:08.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:08.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:08.181 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 66.212692247000007, 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:52:08.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.186 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 66.212629301000007, 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:52:08.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 66.212595618999998, 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:08.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:52:08.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:08.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:08.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:08.196 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:08.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:08.198 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:08.199 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:08.197+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:08.200 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:08.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:08.201 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:08.202 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:08.202 INFO:tasks.rados.rados.0.smithi159.stdout:1808: done (10 left) 2024-09-13T13:52:08.202 INFO:tasks.rados.rados.0.smithi159.stdout:1809: done (9 left) 2024-09-13T13:52:08.202 INFO:tasks.rados.rados.0.smithi159.stdout:1810: done (8 left) 2024-09-13T13:52:08.202 INFO:tasks.rados.rados.0.smithi159.stdout:1811: done (7 left) 2024-09-13T13:52:08.202 INFO:tasks.rados.rados.0.smithi159.stdout:1812: done (6 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1813: done (5 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1814: done (4 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1815: done (3 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1816: done (2 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1817: done (1 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1818: done (0 left) 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: write (excl) oid 17 current snap is 49 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: writing smithi15938319-17 from 648836 to 1271373 tid 1 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: writing smithi15938319-17 from 1981263 to 2474374 tid 2 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: writing smithi15938319-17 from 3107319 to 3848836 tid 3 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1820: write oid 50 current snap is 49 2024-09-13T13:52:08.203 INFO:tasks.rados.rados.0.smithi159.stdout:1819: finishing write tid 1 to smithi15938319-17 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1819: finishing write tid 2 to smithi15938319-17 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1820: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1820: writing smithi15938319-50 from 665643 to 1363429 tid 1 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1820: writing smithi15938319-50 from 2039745 to 2792867 tid 2 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1820: writing smithi15938319-50 from 3534742 to 3865643 tid 3 2024-09-13T13:52:08.204 INFO:tasks.rados.rados.0.smithi159.stdout:1821: snap_remove snap 40 2024-09-13T13:52:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:08.204 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:08.205 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:08.205 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:08.212 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:08.212 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:08.234 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:08.234 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:08.271 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:08.271 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:08.299 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:08.298+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:08.307 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:08.307 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:08.309 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.309 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:08.328 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:08.332+0000 7f39e5d9b640 0 -- 172.21.15.27:0/2626285667 >> v2:172.21.15.159:6805/4073091144 conn(0x7f39e8222980 msgr2=0x7f39e8224d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:08.356 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:08.356 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:08.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:08.402 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:08.401+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:08.460 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.460 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.468 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:08.496 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:08.494+0000 7f9b0c553640 -1 osd.12 144 handle_osd_map injecting map crc failure 2024-09-13T13:52:08.500 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:08.499+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:08.601 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:08.601+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:08.612 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.613 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:08.703 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:08.702+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:08.722 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:08.722+0000 7fc16a180640 -1 osd.11 144 handle_osd_map injecting map crc failure 2024-09-13T13:52:08.724 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.724 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:08.742 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:08.746 INFO:tasks.rados.rados.0.smithi159.stdout:1821: done (2 left) 2024-09-13T13:52:08.746 INFO:tasks.rados.rados.0.smithi159.stdout:1822: rollback oid 36 current snap is 49 2024-09-13T13:52:08.746 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 36 to 32 2024-09-13T13:52:08.746 INFO:tasks.rados.rados.0.smithi159.stdout:1819: finishing write tid 3 to smithi15938319-17 2024-09-13T13:52:08.746 INFO:tasks.rados.rados.0.smithi159.stdout:1819: finishing write tid 4 to smithi15938319-17 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: finishing write tid 5 to smithi15938319-17 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: oid 17 updating version 0 to 392 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: oid 17 updating version 392 to 393 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: oid 17 updating version 393 to 394 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: oid 17 updating version 394 to 395 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 395 (ObjNum 348 snap 49 seq_num 348) dirty exists 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1819: left oid 17 (ObjNum 348 snap 49 seq_num 348) 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: finishing write tid 1 to smithi15938319-50 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: finishing write tid 2 to smithi15938319-50 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: finishing write tid 3 to smithi15938319-50 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: finishing write tid 4 to smithi15938319-50 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: finishing write tid 5 to smithi15938319-50 2024-09-13T13:52:08.747 INFO:tasks.rados.rados.0.smithi159.stdout:1820: oid 50 updating version 0 to 287 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1820: oid 50 version 287 is already newer than 286 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1820: oid 50 updating version 287 to 289 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1820: oid 50 version 289 is already newer than 288 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 50 v 289 (ObjNum 349 snap 49 seq_num 349) dirty exists 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1820: left oid 50 (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1819: done (2 left) 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1820: done (1 left) 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1823: read oid 40 snap 34 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1823: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1824: tier_evict oid 32 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1824: tier_evict oid 32 snap -1 2024-09-13T13:52:08.748 INFO:tasks.rados.rados.0.smithi159.stdout:1825: snap_create 2024-09-13T13:52:08.749 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:08.745+0000 7f26bcfe2640 -1 osd.3 145 handle_osd_map injecting map crc failure 2024-09-13T13:52:08.750 INFO:tasks.rados.rados.0.smithi159.stdout:1824: got (22) Invalid argument 2024-09-13T13:52:08.754 INFO:tasks.rados.rados.0.smithi159.stdout:1823: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:08.804 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:08.802+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:08.809 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 71454 2024-09-13T13:52:08.839 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.839 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:08.895 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2900755 2024-09-13T13:52:08.901 INFO:tasks.rados.rados.0.smithi159.stdout:1822: finishing rollback tid 0 to smithi15938319-36 2024-09-13T13:52:08.901 INFO:tasks.rados.rados.0.smithi159.stdout:1822: finishing rollback tid 1 to smithi15938319-36 2024-09-13T13:52:08.901 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 277 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-13T13:52:08.904 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:08.904+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:08.949 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:08.950 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:08.958 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:09.005 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:09.004+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:09.062 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.062 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.070 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:09.106 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:09.105+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:09.174 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.174 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:09.206 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:09.205+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:09.286 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.286 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.295 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:09.307 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:09.307+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:09.398 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.398 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:09.408 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:09.407+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:09.509 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:09.508+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:09.511 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.511 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.519 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:09.609 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:09.609+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:09.621 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.621 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:09.714 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:09.713+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:09.747 INFO:tasks.rados.rados.0.smithi159.stdout:1822: done (3 left) 2024-09-13T13:52:09.747 INFO:tasks.rados.rados.0.smithi159.stdout:1823: done (2 left) 2024-09-13T13:52:09.747 INFO:tasks.rados.rados.0.smithi159.stdout:1824: done (1 left) 2024-09-13T13:52:09.747 INFO:tasks.rados.rados.0.smithi159.stdout:1825: done (0 left) 2024-09-13T13:52:09.747 INFO:tasks.rados.rados.0.smithi159.stdout:1826: set_chunk oid 18 target oid 2024-09-13T13:52:09.757 INFO:tasks.rados.rados.0.smithi159.stdout:1826: set_chunk oid 18 offset: 8704 length: 120320 target oid 4d01c7200127ae6415f90a3e5a23797fe4f2ef329c05626cac3dddcdc8709921 offset: 0 snap -1 2024-09-13T13:52:09.757 INFO:tasks.rados.rados.0.smithi159.stdout:1827: read oid 10 snap -1 2024-09-13T13:52:09.757 INFO:tasks.rados.rados.0.smithi159.stdout:1827: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:09.758 INFO:tasks.rados.rados.0.smithi159.stdout:1828: write (excl) oid 3 current snap is 50 2024-09-13T13:52:09.758 INFO:tasks.rados.rados.0.smithi159.stdout:1828: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-13T13:52:09.763 INFO:tasks.rados.rados.0.smithi159.stdout:1828: writing smithi15938319-3 from 682450 to 1171838 tid 1 2024-09-13T13:52:09.769 INFO:tasks.rados.rados.0.smithi159.stdout:1828: writing smithi15938319-3 from 1814580 to 2311360 tid 2 2024-09-13T13:52:09.776 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.776 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.777 INFO:tasks.rados.rados.0.smithi159.stdout:1828: writing smithi15938319-3 from 2878518 to 3666546 tid 3 2024-09-13T13:52:09.777 INFO:tasks.rados.rados.0.smithi159.stdout:1829: set_chunk oid 21 target oid 2024-09-13T13:52:09.783 INFO:tasks.rados.rados.0.smithi159.stdout:1829: set_chunk oid 21 offset: 1777152 length: 91648 target oid e1b8e26fc62ce79fb3b85c40dee80abe9984ac99e01af9de4048afb0abe73bf7 offset: 0 snap 46 2024-09-13T13:52:09.783 INFO:tasks.rados.rados.0.smithi159.stdout:1830: tier_promote oid 41 2024-09-13T13:52:09.783 INFO:tasks.rados.rados.0.smithi159.stdout:1826: finishing set_chunk to oid 18 2024-09-13T13:52:09.783 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 219 (ObjNum 266 snap 32 seq_num 266) dirty exists 2024-09-13T13:52:09.783 INFO:tasks.rados.rados.0.smithi159.stdout:1827: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:09.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:09.827 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:09.826+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1828: finishing write tid 1 to smithi15938319-3 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1828: finishing write tid 2 to smithi15938319-3 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1826: done (4 left) 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1827: done (3 left) 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1831: write oid 13 current snap is 50 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1828: finishing write tid 3 to smithi15938319-3 2024-09-13T13:52:09.910 INFO:tasks.rados.rados.0.smithi159.stdout:1831: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-13T13:52:09.913 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:09.912+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:09.917 INFO:tasks.rados.rados.0.smithi159.stdout:1831: writing smithi15938319-13 from 699257 to 1263894 tid 1 2024-09-13T13:52:09.928 INFO:tasks.rados.rados.0.smithi159.stdout:1831: writing smithi15938319-13 from 1873062 to 2513500 tid 2 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1831: writing smithi15938319-13 from 3305941 to 3899257 tid 3 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1832: tier_promote oid 49 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: finishing write tid 4 to smithi15938319-3 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: finishing write tid 5 to smithi15938319-3 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: oid 3 updating version 0 to 330 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: oid 3 version 330 is already newer than 328 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: oid 3 updating version 330 to 331 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: oid 3 version 331 is already newer than 329 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 331 (ObjNum 350 snap 50 seq_num 350) dirty exists 2024-09-13T13:52:09.936 INFO:tasks.rados.rados.0.smithi159.stdout:1828: left oid 3 (ObjNum 350 snap 50 seq_num 350) 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1829: finishing set_chunk to oid 21 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1830: got (0) Success 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 279 (ObjNum 337 snap 47 seq_num 337) dirty exists 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1831: finishing write tid 1 to smithi15938319-13 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1828: done (4 left) 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1829: done (3 left) 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1830: done (2 left) 2024-09-13T13:52:09.937 INFO:tasks.rados.rados.0.smithi159.stdout:1833: set_chunk oid 15 target oid 2024-09-13T13:52:09.942 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:09.942 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:09.944 INFO:tasks.rados.rados.0.smithi159.stdout:1833: set_chunk oid 15 offset: 1515520 length: 580608 target oid e60242d0e1d7ed9a8edf57238cc6473181e3d2f8bdc2387a0ea1b55f64be5426 offset: 0 snap -1 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1834: read oid 26 snap 48 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1834: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1835: tier_evict oid 43 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1835: tier_evict oid 43 snap -1 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: finishing write tid 2 to smithi15938319-13 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1836: tier_promote oid 34 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1832: got (0) Success 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 76 (ObjNum 332 snap 47 seq_num 332) dirty exists 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: finishing write tid 3 to smithi15938319-13 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: finishing write tid 4 to smithi15938319-13 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: finishing write tid 5 to smithi15938319-13 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: oid 13 updating version 0 to 176 2024-09-13T13:52:09.945 INFO:tasks.rados.rados.0.smithi159.stdout:1831: oid 13 updating version 176 to 179 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1831: oid 13 version 179 is already newer than 177 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1831: oid 13 version 179 is already newer than 178 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 179 (ObjNum 351 snap 50 seq_num 351) dirty exists 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1831: left oid 13 (ObjNum 351 snap 50 seq_num 351) 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1831: done (5 left) 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1832: done (4 left) 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1837: read oid 39 snap -1 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1837: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1838: read oid 22 snap -1 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1838: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:52:09.946 INFO:tasks.rados.rados.0.smithi159.stdout:1839: set_chunk oid 24 target oid 2024-09-13T13:52:09.955 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:09.961 INFO:tasks.rados.rados.0.smithi159.stdout:1839: set_chunk oid 24 offset: 68096 length: 1619968 target oid c6762ecaf7afcfc8ceb406eeff773ec7d2f717f7f00ae1ce416fb44256527385 offset: 0 snap -1 2024-09-13T13:52:09.961 INFO:tasks.rados.rados.0.smithi159.stdout:1840: read oid 25 snap -1 2024-09-13T13:52:09.961 INFO:tasks.rados.rados.0.smithi159.stdout:1840: expect (ObjNum 331 snap 47 seq_num 331) 2024-09-13T13:52:09.961 INFO:tasks.rados.rados.0.smithi159.stdout:1841: read oid 33 snap -1 2024-09-13T13:52:09.962 INFO:tasks.rados.rados.0.smithi159.stdout:1841: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:52:09.962 INFO:tasks.rados.rados.0.smithi159.stdout:1842: set_chunk oid 18 target oid 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1842: set_chunk oid 18 offset: 312320 length: 302592 target oid 32915e59233feaf52c4df716f80f6852b7ef01008a610afa1022c2a2de413908 offset: 0 snap -1 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1843: read oid 7 snap 38 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1843: expect (ObjNum 284 snap 38 seq_num 284) 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1844: tier_evict oid 23 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1833: finishing set_chunk to oid 15 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 1515520 length: 580608 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1844: tier_evict oid 23 snap -1 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1835: got (22) Invalid argument 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1833: done (11 left) 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1835: done (10 left) 2024-09-13T13:52:09.967 INFO:tasks.rados.rados.0.smithi159.stdout:1845: read oid 42 snap 46 2024-09-13T13:52:09.968 INFO:tasks.rados.rados.0.smithi159.stdout:1845: expect (ObjNum 327 snap 46 seq_num 327) 2024-09-13T13:52:09.968 INFO:tasks.rados.rados.0.smithi159.stdout:1846: write oid 8 current snap is 50 2024-09-13T13:52:09.968 INFO:tasks.rados.rados.0.smithi159.stdout:1836: got (0) Success 2024-09-13T13:52:09.968 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 235 (ObjNum 323 snap 46 seq_num 323) dirty exists 2024-09-13T13:52:09.968 INFO:tasks.rados.rados.0.smithi159.stdout:1838: expect (ObjNum 304 snap 43 seq_num 304) 2024-09-13T13:52:09.992 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2530401 2024-09-13T13:52:10.014 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:10.013+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:10.040 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2162739 2024-09-13T13:52:10.063 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:10.064 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:10.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:10.114 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:10.113+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:10.120 INFO:tasks.rados.rados.0.smithi159.stdout:1834: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:10.129 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1368407 2024-09-13T13:52:10.161 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:10.215 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:10.214+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:10.218 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:10.218 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:10.227 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:10.255 INFO:tasks.rados.rados.0.smithi159.stdout:1837: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:10.315 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:10.315+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:10.322 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:10.359 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:10.360 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:10.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:10.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:10.416 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:10.415+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:10.465 INFO:tasks.rados.rados.0.smithi159.stdout:1841: expect (ObjNum 278 snap 38 seq_num 278) 2024-09-13T13:52:10.477 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2194277 2024-09-13T13:52:10.509 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:10.509 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:10.512 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:10.512 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:10.517 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:10.516+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:10.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:10.527 INFO:tasks.rados.rados.0.smithi159.stdout:1839: finishing set_chunk to oid 24 2024-09-13T13:52:10.527 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 24 v 278 (ObjNum 339 snap 47 seq_num 339) dirty exists 2024-09-13T13:52:10.527 INFO:tasks.rados.rados.0.smithi159.stdout:1846: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-13T13:52:10.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:10.533 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:10.534 INFO:tasks.rados.rados.0.smithi159.stdout:1846: writing smithi15938319-8 from 716064 to 1355950 tid 1 2024-09-13T13:52:10.539 INFO:tasks.rados.rados.0.smithi159.stdout:1846: writing smithi15938319-8 from 1815191 to 2315640 tid 2 2024-09-13T13:52:10.545 INFO:tasks.rados.rados.0.smithi159.stdout:1846: writing smithi15938319-8 from 2817011 to 3343936 tid 3 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1834: done (11 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1836: done (10 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1837: done (9 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1838: done (8 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1839: done (7 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1841: done (6 left) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1847: read oid 11 snap 49 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1847: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1848: delete oid 22 current snap is 50 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1844: got (22) Invalid argument 2024-09-13T13:52:10.546 INFO:tasks.rados.rados.0.smithi159.stdout:1842: finishing set_chunk to oid 18 2024-09-13T13:52:10.547 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 219 (ObjNum 266 snap 32 seq_num 266) dirty exists 2024-09-13T13:52:10.547 INFO:tasks.rados.rados.0.smithi159.stdout:1843: expect (ObjNum 284 snap 38 seq_num 284) 2024-09-13T13:52:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:10.576 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:10.576 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:10.576 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:10.584 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1945451 2024-09-13T13:52:10.598 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:10.598 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:10.609 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:10.618 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:10.617+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:10.620 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:10.620 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:10.630 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:10.630 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:10.631 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:10.631 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:10.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:10.719 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:10.718+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:10.719 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1587138 2024-09-13T13:52:10.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:10.738 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:10.751 INFO:tasks.rados.rados.0.smithi159.stdout:1845: expect (ObjNum 327 snap 46 seq_num 327) 2024-09-13T13:52:10.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:52:10.759 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:10.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:10.782 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:10.782 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:10.790 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:10.790 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:10.790 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:10.790 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:10.790 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:10.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:10.805 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:10.805 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:10.820 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:10.819+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:10.826 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:10.826 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:10.836 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:10.847 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:10.847 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:10.870 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:10.870 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:10.892 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:10.892 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:10.916 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:10.916 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:10.917 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.562471411, 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.918 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.919 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:52:10.920 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.921 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.922 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.92301922999999, 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.923 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:52:10.924 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.925 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:52:10.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.927 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 105.920422273, 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.928 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:10.929 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.930 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:52:10.931 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 102.18071252999999, 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:10.932 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.933 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.934 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:52:10.935 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 102.16726780499999, 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:52:10.938 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:52:10.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 89.272321091999999, 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:10.940 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:10.944 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 89.228606008, 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:52:10.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 89.034792425000006, 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:52:10.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 79.154400581999994, 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:10.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:52:10.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.939309237000003, 2024-09-13T13:52:10.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.963 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:52:10.964 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.938430808000007, 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:10.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:52:10.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:52:10.969 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.914715310000005, 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:52:10.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:52:10.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 74.141353261000006, 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.974 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:52:10.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 74.140752371999994, 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:52:10.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 74.133599794999995, 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.983 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:52:10.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:52:10.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 73.827835381, 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:10.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:52:10.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:10.992 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 73.820203848999995, 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.993 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:52:10.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 70.108732705999998, 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:10.997 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:52:10.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:52:11.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 70.107797739000006, 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.002 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:52:11.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:52:11.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 70.097114904999998, 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:11.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.011 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.012 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.013 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:11.011+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:11.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:11.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:11.018 INFO:tasks.rados.rados.0.smithi159.stdout:1840: expect (ObjNum 331 snap 47 seq_num 331) 2024-09-13T13:52:11.018 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.113 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:11.112+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:11.114 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 74107 2024-09-13T13:52:11.162 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.162 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:11.162 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:11.163 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:11.210 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.214 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:11.213+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1846: finishing write tid 1 to smithi15938319-8 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1846: finishing write tid 2 to smithi15938319-8 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1840: done (7 left) 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1842: done (6 left) 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1843: done (5 left) 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1844: done (4 left) 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1845: done (3 left) 2024-09-13T13:52:11.307 INFO:tasks.rados.rados.0.smithi159.stdout:1848: done (2 left) 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1849: read oid 2 snap -1 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1849: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1850: read oid 47 snap -1 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1850: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1851: write oid 13 current snap is 50 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1846: finishing write tid 3 to smithi15938319-8 2024-09-13T13:52:11.308 INFO:tasks.rados.rados.0.smithi159.stdout:1851: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-13T13:52:11.315 INFO:tasks.rados.rados.0.smithi159.stdout:1851: writing smithi15938319-13 from 749678 to 1540062 tid 1 2024-09-13T13:52:11.315 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:11.314+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:11.320 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:11.320 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:11.320 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:11.321 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.321 INFO:tasks.rados.rados.0.smithi159.stdout:1851: writing smithi15938319-13 from 2332155 to 2836273 tid 2 2024-09-13T13:52:11.327 INFO:tasks.rados.rados.0.smithi159.stdout:1851: writing smithi15938319-13 from 3271857 to 3821326 tid 3 2024-09-13T13:52:11.327 INFO:tasks.rados.rados.0.smithi159.stdout:1852: set_chunk oid 14 target oid 2024-09-13T13:52:11.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1852: set_chunk oid 14 offset: 1788928 length: 463360 target oid 0d01361f24984ff4b0b1eb9d9ead6642f5b878662a350c2f914a3a82940a2cbd offset: 0 snap -1 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1853: rollback oid 37 current snap is 50 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 37 to 33 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: finishing write tid 4 to smithi15938319-8 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: finishing write tid 5 to smithi15938319-8 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: oid 8 updating version 0 to 209 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: oid 8 updating version 209 to 212 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: oid 8 version 212 is already newer than 211 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:1846: oid 8 version 212 is already newer than 210 2024-09-13T13:52:11.339 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 212 (ObjNum 352 snap 50 seq_num 352) dirty exists 2024-09-13T13:52:11.340 INFO:tasks.rados.rados.0.smithi159.stdout:1846: left oid 8 (ObjNum 352 snap 50 seq_num 352) 2024-09-13T13:52:11.340 INFO:tasks.rados.rados.0.smithi159.stdout:1847: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:11.340 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.395 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.416 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:11.415+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:11.439 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.440 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 119.27163023599999, 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:52:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.443 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.444 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.446 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.447 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 119.271604243, 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.448 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:11.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:52:11.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:52:11.453 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 119.166295421, 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:52:11.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.87026024399999, 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:52:11.458 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:52:11.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.464 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.870245744, 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:52:11.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:52:11.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:52:11.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.867992304, 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:52:11.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:52:11.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.473 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.745569016, 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:52:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:52:11.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:52:11.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.745538197, 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.480 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:52:11.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.743380089, 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:11.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:52:11.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 118.533845891, 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:52:11.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:52:11.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 117.667868359, 2024-09-13T13:52:11.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:11.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:52:11.503 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 117.65419264499999, 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:52:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.638291061000004, 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:52:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:52:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.629999565999995, 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.516 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:52:11.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:52:11.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:52:11.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:11.524 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.625972645000004, 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:52:11.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:52:11.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.625647689000004, 2024-09-13T13:52:11.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.535 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:52:11.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.625616163999993, 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.540 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:52:11.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:52:11.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:52:11.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 66.597926162999997, 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:52:11.547 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:52:11.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3255 4.a 4:5004af91:::smithi15938319-34:head [append 0~0] snapc 23=[23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 42.689343305999998, 2024-09-13T13:52:11.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.285444431, 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3255 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:11.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747900+0000", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1220000000000002e-06 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747906+0000", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5979999999999996e-06 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747911+0000", 2024-09-13T13:52:11.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4e-06 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750840+0000", 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029289250000000002 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750936+0000", 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5449e-05 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751081+0000", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014517800000000001 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751499+0000", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00041844799999999998 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033276+0000", 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28177714300000001 2024-09-13T13:52:11.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033312+0000", 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5370999999999997e-05 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033342+0000", 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0457999999999999e-05 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:3255 4.a e112/85 ondisk, result = 0)", 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 42.684704212, 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.280808787, 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752540+0000", 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9849999999999998e-06 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033199+0000", 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28065851600000002 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033203+0000", 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5029999999999999e-06 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033267+0000", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3596999999999994e-05 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033273+0000", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.789e-06 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033346+0000", 2024-09-13T13:52:11.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2960999999999995e-05 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.562 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.563 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:11.562+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:11.564 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2308675 2024-09-13T13:52:11.564 INFO:tasks.rados.rados.0.smithi159.stdout:1850: expect (ObjNum 262 snap 30 seq_num 262) 2024-09-13T13:52:11.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:11.604 INFO:tasks.rados.rados.0.smithi159.stdout:1851: finishing write tid 1 to smithi15938319-13 2024-09-13T13:52:11.604 INFO:tasks.rados.rados.0.smithi159.stdout:1849: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:11.604 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:11.665 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:11.664+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:11.672 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.672 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:11.672 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:11.672 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:11.747 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:11.769 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:11.767+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:11.790 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.791 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:11.791 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:11.791 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:11.791 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:11.799 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1851: finishing write tid 2 to smithi15938319-13 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1851: finishing write tid 3 to smithi15938319-13 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1846: done (6 left) 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1847: done (5 left) 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1849: done (4 left) 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1850: done (3 left) 2024-09-13T13:52:11.831 INFO:tasks.rados.rados.0.smithi159.stdout:1854: read oid 11 snap -1 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1854: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1855: tier_promote oid 32 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: finishing write tid 4 to smithi15938319-13 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: finishing write tid 5 to smithi15938319-13 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: oid 13 updating version 0 to 182 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: oid 13 version 182 is already newer than 181 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: oid 13 version 182 is already newer than 180 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: oid 13 updating version 182 to 183 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 183 (ObjNum 354 snap 50 seq_num 354) dirty exists 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1851: left oid 13 (ObjNum 354 snap 50 seq_num 354) 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1853: finishing rollback tid 0 to smithi15938319-37 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:1853: finishing rollback tid 1 to smithi15938319-37 2024-09-13T13:52:11.832 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 249 (ObjNum 273 snap 0 seq_num 2476711392) dirty dne 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1852: finishing set_chunk to oid 14 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 204 (ObjNum 315 snap 45 seq_num 315) dirty exists 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1851: done (4 left) 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1852: done (3 left) 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1853: done (2 left) 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1856: read oid 1 snap -1 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1856: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1857: read oid 37 snap -1 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1857: expect deleted 2024-09-13T13:52:11.833 INFO:tasks.rados.rados.0.smithi159.stdout:1858: set_chunk oid 48 target oid 2024-09-13T13:52:11.842 INFO:tasks.rados.rados.0.smithi159.stdout:1858: set_chunk oid 48 offset: 1749504 length: 1126400 target oid 1ecf2ad9d190a4f053560b6e50f2e285433a1acdc8f32561912f4f1378a81b60 offset: 0 snap -1 2024-09-13T13:52:11.842 INFO:tasks.rados.rados.0.smithi159.stdout:1859: tier_flush oid 13 2024-09-13T13:52:11.843 INFO:tasks.rados.rados.0.smithi159.stdout:1859: tier_flush oid 13 snap -1 2024-09-13T13:52:11.843 INFO:tasks.rados.rados.0.smithi159.stdout:1855: got (0) Success 2024-09-13T13:52:11.843 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 171 (ObjNum 346 snap 48 seq_num 346) dirty exists 2024-09-13T13:52:11.843 INFO:tasks.rados.rados.0.smithi159.stdout:1854: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:11.870 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:11.869+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:11.875 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1977702 2024-09-13T13:52:11.913 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1680298 2024-09-13T13:52:11.948 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:11.948 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:11.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:11.948 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:11.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 119.04792132199999, 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.949 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:52:11.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:11.955 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 119.047853465, 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.956 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:11.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:52:11.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:52:11.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:52:11.962 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717878853, 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.963 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:11.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:52:11.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717844378, 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:52:11.969 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:52:11.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:52:11.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:52:11.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:52:11.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717829392, 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.977 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:52:11.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717815343, 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.982 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.984 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:52:11.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:11.987 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717793628, 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:52:11.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.717761783, 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:11.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:52:11.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:11.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.997 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.556065057, 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:11.998 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:11.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:12.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:52:12.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.556031907, 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.004 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:52:12.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:52:12.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.556001689, 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:12.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:52:12.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.015 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.55595934, 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:52:12.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:52:12.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.55592512299999, 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:52:12.022 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:52:12.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:52:12.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:12.028 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.555891852, 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:52:12.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:52:12.034 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 103.153608619, 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:52:12.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 69.128542010000004, 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.039 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:52:12.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:52:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:52:12.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:52:12.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2965 4.c 4:358f784f:::smithi15938319-41:head [append 0~0] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:12.046 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 55.072463487, 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014521253, 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2965 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:12.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873612+0000", 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2999999999999999e-07 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873613+0000", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.398e-06 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873616+0000", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7190000000000001e-06 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873618+0000", 2024-09-13T13:52:12.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6170000000000001e-06 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873745+0000", 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126389 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873747+0000", 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4030000000000001e-06 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882806+0000", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090587889999999994 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882861+0000", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4793999999999997e-05 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882955+0000", 2024-09-13T13:52:12.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4377999999999997e-05 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883725+0000", 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076983499999999996 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888075+0000", 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043496630000000001 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888105+0000", 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0735000000000003e-05 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888133+0000", 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7302999999999999e-05 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:4008 4.c e138/52 ondisk, result = 0)", 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 11.856292271000001, 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95524980500000001, 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2799999999999996e-07 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089787+0000", 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.422e-06 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044661+0000", 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95487436400000003 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044676+0000", 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4746000000000001e-05 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044804+0000", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012786699999999999 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044823+0000", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9296e-05 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.045033+0000", 2024-09-13T13:52:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00020948200000000001 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4104 4.c 4:358f784f:::smithi15938319-41:head [write 463959~478445,stat] snapc 30=[30,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 8.8602558889999994, 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013243293, 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.058 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4104 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.059 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085821+0000", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.199e-06 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086649+0000", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082724800000000005 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086672+0000", 2024-09-13T13:52:12.060 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3768e-05 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086684+0000", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1584e-05 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086762+0000", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8566000000000006e-05 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086770+0000", 2024-09-13T13:52:12.061 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091302+0000", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045321670000000001 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091427+0000", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012461500000000001 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091571+0000", 2024-09-13T13:52:12.062 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014418699999999999 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098930+0000", 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0073584499999999999 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098994+0000", 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4639000000000002e-05 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.063 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099030+0000", 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5627000000000003e-05 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099062+0000", 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4172 4.c 4:34327e96:::smithi15938319-44:head [write 581608~721541] snapc 30=[30,2d,2a,28,25,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:12.064 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 5.8517379390000004, 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012273429000000001, 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4172 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:12.065 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094339+0000", 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0930000000000001e-06 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:12.066 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095120+0000", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000781268 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095139+0000", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8553999999999999e-05 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095147+0000", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2700000000000004e-06 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.067 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095231+0000", 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3923000000000003e-05 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095236+0000", 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7589999999999997e-06 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099704+0000", 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044682710000000002 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.068 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099783+0000", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9190000000000006e-05 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099874+0000", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0242999999999996e-05 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.101994+0000", 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021203110000000002 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106540+0000", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004545922 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106578+0000", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8114999999999997e-05 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106610+0000", 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2509999999999999e-05 2024-09-13T13:52:12.070 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:12.071 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:12.071 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:12.072 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2410323 2024-09-13T13:52:12.073 INFO:tasks.rados.rados.0.smithi159.stdout:1856: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:12.073 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2164563 2024-09-13T13:52:12.073 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:12.071+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:12.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:12.103 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:12.171 INFO:tasks.rados.rados.0.smithi159.stdout:1859: got (16) Device or resource busy 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 13 v 183 (ObjNum 354 snap 50 seq_num 354) unflushed 50 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1858: finishing set_chunk to oid 48 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 1749504 length: 1126400 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 0 (ObjNum 319 snap 46 seq_num 319) dirty exists 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1854: done (5 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1855: done (4 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1856: done (3 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1857: done (2 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1858: done (1 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1859: done (0 left) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1860: read oid 7 snap -1 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1860: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:52:12.172 INFO:tasks.rados.rados.0.smithi159.stdout:1861: snap_remove snap 38 2024-09-13T13:52:12.176 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:12.173+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:12.189 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.190 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:12.190 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:12.190 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:12.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:12.274 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:12.273+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:12.302 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:12.302 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:12.302 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:12.310 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:12.375 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:12.374+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:12.415 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 119.497955945, 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:52:12.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 119.49315641600001, 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:52:12.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:52:12.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:52:12.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 118.643808378, 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:52:12.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.139510778, 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:52:12.434 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:52:12.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:52:12.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.133650829, 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.442 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:52:12.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:52:12.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:52:12.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.128706188, 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:52:12.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:52:12.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:52:12.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.12849561, 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:52:12.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:52:12.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:52:12.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.128447511, 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:52:12.466 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:52:12.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:52:12.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.04536710000001, 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:52:12.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.478 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.045258389, 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.483 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.04514310899999, 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1796 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.171172006999996, 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094313859999999999, 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1796 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242075+0000", 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4619999999999999e-06 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242087+0000", 2024-09-13T13:52:12.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.046e-05 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242112+0000", 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4575000000000001e-05 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242124+0000", 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2014999999999999e-05 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242242+0000", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117871 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242251+0000", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6799999999999999e-06 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251344+0000", 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090938249999999998 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251506+0000", 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016149799999999999 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1797 4.7 4:e63fe140:::smithi15938319-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.495 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.171085689999998, 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094858990000000008, 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1797 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.496 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242161+0000", 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242162+0000", 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6300000000000004e-07 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242167+0000", 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1410000000000004e-06 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242177+0000", 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0597999999999999e-05 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242184+0000", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0899999999999999e-06 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242340+0000", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015548100000000001 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242346+0000", 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9449999999999998e-06 2024-09-13T13:52:12.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251540+0000", 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091938709999999993 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251647+0000", 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010700999999999999 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1798 4.7 4:e63fe140:::smithi15938319-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.171025319999998, 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0095840660000000005, 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1798 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.501 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242221+0000", 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4600000000000004e-07 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242229+0000", 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8870000000000003e-06 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1740000000000006e-06 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242245+0000", 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015256499999999999 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242404+0000", 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1510000000000001e-06 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251681+0000", 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0092774559999999999 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251805+0000", 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012421200000000001 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.505 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 70.537173175000007, 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.506 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:52:12.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.512 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:3134 4.7 4:e1bd516b:::smithi15938319-19:head [write 769820~720742,stat] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.240332571000003, 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096203899999999995, 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 3134 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172914+0000", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.172917+0000", 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6759999999999999e-06 2024-09-13T13:52:12.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173729+0000", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00081220499999999998 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173743+0000", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4073000000000001e-05 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173749+0000", 2024-09-13T13:52:12.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173849+0000", 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9809999999999995e-05 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.173917+0000", 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7556000000000002e-05 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.174067+0000", 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000150335 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.180026+0000", 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059590329999999999 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182455+0000", 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024288000000000001 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182498+0000", 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2861000000000002e-05 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:22.182534+0000", 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6522e-05 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.518 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(osd.1.0:130 4.7 e130/52 ondisk, result = 0)", 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 20.598955736000001, 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18760375600000001, 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814292+0000", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7999999999999995e-07 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814296+0000", 2024-09-13T13:52:12.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8339999999999998e-06 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001621+0000", 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18732465400000001 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001634+0000", 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2883e-05 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001774+0000", 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000140104 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001805+0000", 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001895+0000", 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9456000000000005e-05 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4040 4.7 4:e26a40dd:::smithi15938319-25:head [write 763117~710598] snapc 30=[30,2f,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 11.285227889, 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010818598, 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4040 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128020+0000", 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.778e-06 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128710+0000", 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000689637 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128727+0000", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6628000000000001e-05 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128734+0000", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128785+0000", 2024-09-13T13:52:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0828999999999997e-05 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128789+0000", 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1409999999999998e-06 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133000+0000", 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042116080000000004 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133067+0000", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6644999999999996e-05 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133188+0000", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121249 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138729+0000", 2024-09-13T13:52:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055408610000000002 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138749+0000", 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0029000000000001e-05 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138796+0000", 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6681999999999998e-05 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138837+0000", 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1440000000000003e-05 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4225 4.7 4:e26a40dd:::smithi15938319-25:23 [read 0~3630279,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0884956470000002, 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010120242, 2024-09-13T13:52:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4225 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8000000000000001e-07 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324754+0000", 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6419999999999999e-06 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324758+0000", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6629999999999999e-06 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324761+0000", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.145e-06 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324877+0000", 2024-09-13T13:52:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116599 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324884+0000", 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4520000000000002e-06 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329877+0000", 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049932090000000002 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329973+0000", 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5606000000000006e-05 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334871+0000", 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048985460000000002 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4226 4.7 4:e26a40dd:::smithi15938319-25:23 [sparse-read 844183~2211810] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0884643729999999, 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011467577, 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4226 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324783+0000", 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324784+0000", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.731e-06 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324789+0000", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324796+0000", 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.537e-06 2024-09-13T13:52:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324912+0000", 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115145 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324918+0000", 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6100000000000002e-06 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334923+0000", 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010004569 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334990+0000", 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7143000000000004e-05 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.336250+0000", 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012599320000000001 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.541 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:12.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:12.543 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:12.483+0000 7f586a715640 -1 osd.9 148 handle_osd_map injecting map crc failure 2024-09-13T13:52:12.543 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:12.541+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:12.580 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:12.578+0000 7f9b0c553640 -1 osd.12 148 handle_osd_map injecting map crc failure 2024-09-13T13:52:12.643 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:12.642+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:12.646 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.646 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:12.646 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:12.646 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:12.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:12.744 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:12.743+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:12.749 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:12.748+0000 7f26bcfe2640 -1 osd.3 149 handle_osd_map injecting map crc failure 2024-09-13T13:52:12.752 INFO:tasks.rados.rados.0.smithi159.stdout:1861: done (1 left) 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1862: read oid 20 snap -1 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1862: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1863: read oid 11 snap -1 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1863: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1864: snap_create 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:1860: expect (ObjNum 294 snap 41 seq_num 294) 2024-09-13T13:52:12.753 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:12.760 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:12.759+0000 7fe0f9e03640 -1 osd.2 149 handle_osd_map injecting map crc failure 2024-09-13T13:52:12.801 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.801 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:12.802 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:12.802 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:12.802 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:12.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:12.845 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:12.844+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 119.143843859, 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:52:12.915 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:12.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:52:12.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 119.134813427, 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:12.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:52:12.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:52:12.926 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.48280755099999, 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:52:12.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:12.931 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.479773565, 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.935 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.603941446999997, 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:12.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:52:12.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:52:12.940 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.598034615000003, 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:12.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:52:12.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:52:12.945 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 97.590599179999998, 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:52:12.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.671305438000005, 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:52:12.950 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:52:12.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:12.956 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.671236773000004, 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:52:12.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:12.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:12.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.671186659, 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.963 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:52:12.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:52:12.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.650229297999999, 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:52:12.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:52:12.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.644633095000003, 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:52:12.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:52:12.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.643089184999994, 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:12.982 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:52:12.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.643024136999998, 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:12.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.993 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:12.994 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 91.036082538000002, 2024-09-13T13:52:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:52:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:12.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:12.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:12.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:52:13.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.180406834999999, 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:13.001 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:52:13.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:52:13.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:52:13.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.108511147000002, 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:13.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:13.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:13.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:52:13.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:13.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.108473626000006, 2024-09-13T13:52:13.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:13.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:52:13.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:52:13.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 80.995554870999996, 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:13.020 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:13.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:52:13.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 76.781488463000002, 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:52:13.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:13.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:52:13.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:13.034 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:13.034 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:13.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:13.036 INFO:tasks.rados.rados.0.smithi159.stdout:1863: expect (ObjNum 272 snap 36 seq_num 272) 2024-09-13T13:52:13.036 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:13.036 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:13.034+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:13.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:13.136 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:13.134+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:13.136 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1244194 2024-09-13T13:52:13.141 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:13.141 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:13.141 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:13.141 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:13.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:13.168 INFO:tasks.rados.rados.0.smithi159.stdout:1862: expect (ObjNum 271 snap 36 seq_num 271) 2024-09-13T13:52:13.174 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:13.196 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:13.218 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:13.224 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:13.237 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:13.236+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:13.239 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:13.240 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:13.240 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:13.261 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:13.262 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:13.280 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:13.284 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:13.284 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:13.292 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:13.292 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:13.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:13.304 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:13.305 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:13.325 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:13.325 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:13.338 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:13.337+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:13.346 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:13.367 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:13.368 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:13.389 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:13.389 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:13.403 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:13.403 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:13.409 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:13.412 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:13.431 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:13.439 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:13.438+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:13.452 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:13.457+0000 7f856259c640 0 -- 172.21.15.27:0/2782340876 >> v2:172.21.15.159:6805/4073091144 conn(0x7f856421fd80 msgr2=0x7f8564222170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:13.479 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:13.479 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:13.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:13.516 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:13.516 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:13.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:13.568 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:13.567+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:13.626 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:13.626 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:13.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:13.676 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:13.675+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:13.741 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:13.740+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:13.745 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:13.743+0000 7f3c37938640 -1 osd.5 150 handle_osd_map injecting map crc failure 2024-09-13T13:52:13.749 INFO:tasks.rados.rados.0.smithi159.stdout:1860: done (3 left) 2024-09-13T13:52:13.749 INFO:tasks.rados.rados.0.smithi159.stdout:1862: done (2 left) 2024-09-13T13:52:13.749 INFO:tasks.rados.rados.0.smithi159.stdout:1863: done (1 left) 2024-09-13T13:52:13.750 INFO:tasks.rados.rados.0.smithi159.stdout:1864: done (0 left) 2024-09-13T13:52:13.750 INFO:tasks.rados.rados.0.smithi159.stdout:1865: rollback oid 47 current snap is 51 2024-09-13T13:52:13.750 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 47 to 35 2024-09-13T13:52:13.750 INFO:tasks.rados.rados.0.smithi159.stdout:1866: delete oid 48 current snap is 51 2024-09-13T13:52:13.755 INFO:tasks.rados.rados.0.smithi159.stdout:1865: finishing rollback tid 0 to smithi15938319-47 2024-09-13T13:52:13.756 INFO:tasks.rados.rados.0.smithi159.stdout:1865: finishing rollback tid 1 to smithi15938319-47 2024-09-13T13:52:13.756 INFO:tasks.rados.rados.0.smithi159.stdout:1865: finishing rollback tid 2 to smithi15938319-47 2024-09-13T13:52:13.756 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 275 (ObjNum 262 snap 30 seq_num 262) dirty exists 2024-09-13T13:52:13.759 INFO:tasks.rados.rados.0.smithi159.stdout:1865: done (1 left) 2024-09-13T13:52:13.759 INFO:tasks.rados.rados.0.smithi159.stdout:1866: done (0 left) 2024-09-13T13:52:13.759 INFO:tasks.rados.rados.0.smithi159.stdout:1867: set_chunk oid 6 target oid 2024-09-13T13:52:13.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:52:13.766 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:13.764+0000 7f1a526ee640 -1 osd.13 150 handle_osd_map injecting map crc failure 2024-09-13T13:52:13.770 INFO:tasks.rados.rados.0.smithi159.stdout:1867: set_chunk oid 6 offset: 1648128 length: 2048 target oid accdbdfb8efa48f9137dbac2fafea8f02440d613212919fe8e5976b735fa1496 offset: 0 snap -1 2024-09-13T13:52:13.770 INFO:tasks.rados.rados.0.smithi159.stdout:1868: read oid 43 snap -1 2024-09-13T13:52:13.770 INFO:tasks.rados.rados.0.smithi159.stdout:1868: expect deleted 2024-09-13T13:52:13.771 INFO:tasks.rados.rados.0.smithi159.stdout:1869: snap_remove snap 46 2024-09-13T13:52:13.772 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:13.772+0000 7f002dbdb640 -1 osd.15 150 handle_osd_map injecting map crc failure 2024-09-13T13:52:13.782 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:13.782 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:13.791 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:13.842 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:13.842+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:13.902 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:13.902 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:13.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:13.944 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:13.943+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:14.045 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:14.044+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:14.054 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.054 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:14.145 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:14.145+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:14.167 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.168 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:14.247 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:14.246+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:14.320 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.320 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.329 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:14.347 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:14.346+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:14.432 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.432 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:14.448 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:14.447+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:14.544 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.544 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.548 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:14.547+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:14.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:14.649 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:14.648+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:14.696 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.696 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:14.751 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:14.749+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:14.752 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:52:14.757 INFO:tasks.rados.rados.0.smithi159.stdout:1869: done (2 left) 2024-09-13T13:52:14.757 INFO:tasks.rados.rados.0.smithi159.stdout:1870: write (excl) oid 33 current snap is 51 2024-09-13T13:52:14.757 INFO:tasks.rados.rados.0.smithi159.stdout:1870: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-13T13:52:14.764 INFO:tasks.rados.rados.0.smithi159.stdout:1870: writing smithi15938319-33 from 783292 to 1324174 tid 1 2024-09-13T13:52:14.770 INFO:tasks.rados.rados.0.smithi159.stdout:1870: writing smithi15938319-33 from 1932766 to 2440553 tid 2 2024-09-13T13:52:14.777 INFO:tasks.rados.rados.0.smithi159.stdout:1870: writing smithi15938319-33 from 3210350 to 3782363 tid 3 2024-09-13T13:52:14.777 INFO:tasks.rados.rados.0.smithi159.stdout:1871: snap_create 2024-09-13T13:52:14.778 INFO:tasks.rados.rados.0.smithi159.stdout:1867: finishing set_chunk to oid 6 2024-09-13T13:52:14.778 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 63 (ObjNum 253 snap 28 seq_num 253) dirty exists 2024-09-13T13:52:14.778 INFO:tasks.rados.rados.0.smithi159.stdout:1870: finishing write tid 1 to smithi15938319-33 2024-09-13T13:52:14.778 INFO:tasks.rados.rados.0.smithi159.stdout:1870: finishing write tid 2 to smithi15938319-33 2024-09-13T13:52:14.781 INFO:tasks.rados.rados.0.smithi159.stdout:1870: finishing write tid 3 to smithi15938319-33 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: finishing write tid 4 to smithi15938319-33 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: finishing write tid 5 to smithi15938319-33 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: oid 33 updating version 0 to 297 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: oid 33 updating version 297 to 298 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: oid 33 version 298 is already newer than 296 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: oid 33 version 298 is already newer than 295 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 298 (ObjNum 356 snap 51 seq_num 356) dirty exists 2024-09-13T13:52:14.782 INFO:tasks.rados.rados.0.smithi159.stdout:1870: left oid 33 (ObjNum 356 snap 51 seq_num 356) 2024-09-13T13:52:14.812 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.812 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:14.852 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:14.851+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:14.923 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:14.923 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:14.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:14.976 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:14.975+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:15.034 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.035 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.043 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:15.084 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:15.083+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:15.156 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:15.154+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:15.186 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.186 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:15.257 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:15.256+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:15.297 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.297 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.305 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:15.358 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:15.357+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:15.421 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.422 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.430 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:15.459 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:15.457+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:15.480 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:15.479+0000 7f4ffbd7a640 -1 osd.1 151 handle_osd_map injecting map crc failure 2024-09-13T13:52:15.480 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:15.479+0000 7f586a715640 -1 osd.9 151 handle_osd_map injecting map crc failure 2024-09-13T13:52:15.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.544 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:15.559 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:15.558+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:15.628 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:15.648 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:15.648 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:15.651 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:15.651 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:15.657 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:15.660 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:15.659+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:15.673 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:15.673 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:15.695 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:15.696 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:15.696 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:15.738 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:15.738 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:15.758 INFO:tasks.rados.rados.0.smithi159.stdout:1867: done (3 left) 2024-09-13T13:52:15.758 INFO:tasks.rados.rados.0.smithi159.stdout:1868: done (2 left) 2024-09-13T13:52:15.759 INFO:tasks.rados.rados.0.smithi159.stdout:1870: done (1 left) 2024-09-13T13:52:15.759 INFO:tasks.rados.rados.0.smithi159.stdout:1871: done (0 left) 2024-09-13T13:52:15.759 INFO:tasks.rados.rados.0.smithi159.stdout:1872: delete oid 49 current snap is 52 2024-09-13T13:52:15.760 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:15.760 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:15.759+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:15.762 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:15.762 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:15.762 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:15.762 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:15.763 INFO:tasks.rados.rados.0.smithi159.stdout:1872: done (0 left) 2024-09-13T13:52:15.764 INFO:tasks.rados.rados.0.smithi159.stdout:1873: read oid 39 snap -1 2024-09-13T13:52:15.764 INFO:tasks.rados.rados.0.smithi159.stdout:1873: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:15.764 INFO:tasks.rados.rados.0.smithi159.stdout:1874: set_chunk oid 19 target oid 2024-09-13T13:52:15.771 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:15.780 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:15.780 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:15.781 INFO:tasks.rados.rados.0.smithi159.stdout:1874: set_chunk oid 19 offset: 162304 length: 1525248 target oid 1ea8cdb3ef8f1d6ea33b2d7fb9ad92cc9973bb39e767fcf9234f3904e0c54ba8 offset: 0 snap 34 2024-09-13T13:52:15.781 INFO:tasks.rados.rados.0.smithi159.stdout:1875: read oid 21 snap -1 2024-09-13T13:52:15.781 INFO:tasks.rados.rados.0.smithi159.stdout:1875: expect (ObjNum 305 snap 43 seq_num 305) 2024-09-13T13:52:15.782 INFO:tasks.rados.rados.0.smithi159.stdout:1876: write (excl) oid 45 current snap is 52 2024-09-13T13:52:15.782 INFO:tasks.rados.rados.0.smithi159.stdout:1876: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-13T13:52:15.785 INFO:tasks.rados.rados.0.smithi159.stdout:1876: writing smithi15938319-45 from 416906 to 824639 tid 1 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1876: writing smithi15938319-45 from 1366083 to 1877539 tid 2 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1877: read oid 5 snap -1 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1877: expect (ObjNum 336 snap 47 seq_num 336) 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1878: read oid 29 snap -1 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1878: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:15.792 INFO:tasks.rados.rados.0.smithi159.stdout:1879: set_chunk oid 15 target oid 2024-09-13T13:52:15.803 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:15.803 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:15.804 INFO:tasks.rados.rados.0.smithi159.stdout:1879: set_chunk oid 15 offset: 631808 length: 590336 target oid 977c059b14621b79fe41b97ef464ca0350b0e1b5a54b8c0f26c0e8ad8bbd82c4 offset: 0 snap -1 2024-09-13T13:52:15.804 INFO:tasks.rados.rados.0.smithi159.stdout:1880: tier_evict oid 25 2024-09-13T13:52:15.804 INFO:tasks.rados.rados.0.smithi159.stdout:1880: tier_evict oid 25 snap -1 2024-09-13T13:52:15.804 INFO:tasks.rados.rados.0.smithi159.stdout:1873: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:15.825 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:15.825 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:15.864 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:15.860+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:15.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:15.909 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:15.909 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:15.962 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:15.961+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:15.995 INFO:tasks.rados.rados.0.smithi159.stdout:1874: finishing set_chunk to oid 19 2024-09-13T13:52:15.995 INFO:tasks.rados.rados.0.smithi159.stdout:1875: expect (ObjNum 305 snap 43 seq_num 305) 2024-09-13T13:52:15.995 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.29682087899999, 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.293844268, 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:52:16.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:52:16.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:52:16.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.291870097, 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.056 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:52:16.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.060 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.253139574, 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:52:16.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:52:16.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:52:16.065 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.249238283, 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:52:16.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:52:16.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:240 3.0 e41/33 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'12, mlcod=41'9)", 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.209200215, 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092645839999999993, 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834516+0000", 2024-09-13T13:52:16.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834518+0000", 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3239999999999999e-06 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835929+0000", 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00141077 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835979+0000", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.994e-05 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.835986+0000", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2989999999999999e-06 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836038+0000", 2024-09-13T13:52:16.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1824999999999999e-05 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.836056+0000", 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8698e-05 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843732+0000", 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076757639999999999 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.843781+0000", 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8964000000000001e-05 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.12807329499999, 2024-09-13T13:52:16.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:16.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:52:16.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.084 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.1232724, 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.085 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:52:16.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.088 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:52:16.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.090 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 121.968633563, 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.091 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:52:16.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:52:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 107.788772077, 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:52:16.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 94.755680721000004, 2024-09-13T13:52:16.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:16.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 80.247040466000001, 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:16.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:52:16.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 54.133013206000001, 2024-09-13T13:52:16.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:16.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:52:16.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:52:16.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:52:16.120 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 53.135410847000003, 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.121 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.123 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:52:16.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4028 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.972869061000001, 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010533003000000001, 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4028 2024-09-13T13:52:16.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070849+0000", 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070861+0000", 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1973999999999999e-05 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070881+0000", 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070894+0000", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2204999999999999e-05 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070965+0000", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1613999999999996e-05 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070971+0000", 2024-09-13T13:52:16.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.395e-06 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077011+0000", 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006040035 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077114+0000", 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103727 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081381+0000", 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042663370000000003 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4029 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.972785017, 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012297313000000001, 2024-09-13T13:52:16.134 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4029 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7400000000000003e-07 2024-09-13T13:52:16.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070935+0000", 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1089999999999998e-06 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070941+0000", 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070945+0000", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2980000000000002e-06 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071034+0000", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8661999999999999e-05 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071041+0000", 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081442+0000", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010401216 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081561+0000", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119207 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083229+0000", 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001667948 2024-09-13T13:52:16.139 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4030 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 14.972763902000001, 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722845, 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.140 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4030 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070955+0000", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5060000000000001e-06 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070958+0000", 2024-09-13T13:52:16.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3160000000000001e-06 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070963+0000", 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8500000000000002e-06 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071060+0000", 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7565999999999994e-05 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071063+0000", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0479999999999999e-06 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083261+0000", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012197846 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083377+0000", 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011555100000000001 2024-09-13T13:52:16.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083676+0000", 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029883399999999999 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4356 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752737+0000", 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.2909794269999999, 2024-09-13T13:52:16.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0093823729999999994, 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4356 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752737+0000", 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752737+0000", 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752739+0000", 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752748+0000", 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8389999999999996e-06 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752771+0000", 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2565000000000002e-05 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752798+0000", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6754000000000002e-05 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752871+0000", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3868000000000005e-05 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752880+0000", 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7420000000000003e-06 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757641+0000", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047603810000000002 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757746+0000", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000105054 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762120+0000", 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0043739759999999999 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4357 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.2908972529999998, 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012624672, 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.151 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4357 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752820+0000", 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1399999999999997e-07 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752827+0000", 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5010000000000001e-06 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752834+0000", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752838+0000", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752943+0000", 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010495199999999999 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752947+0000", 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8789999999999996e-06 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762157+0000", 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092093769999999995 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762260+0000", 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103145 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765444+0000", 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031843850000000001 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4358 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.2908536079999999, 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013684981000000001, 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4358 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752864+0000", 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4700000000000001e-07 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752867+0000", 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4350000000000001e-06 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752874+0000", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3869999999999997e-06 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752880+0000", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752994+0000", 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752999+0000", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0270000000000003e-06 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765566+0000", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012566623000000001 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765652+0000", 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6439000000000004e-05 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.766548+0000", 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000895753 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.162 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:16.164 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:16.162+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: finishing write tid 1 to smithi15938319-45 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: finishing write tid 2 to smithi15938319-45 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: finishing write tid 3 to smithi15938319-45 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: finishing write tid 4 to smithi15938319-45 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: oid 45 updating version 0 to 57 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: oid 45 version 57 is already newer than 55 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: oid 45 version 57 is already newer than 56 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 57 (ObjNum 358 snap 52 seq_num 358) dirty exists 2024-09-13T13:52:16.208 INFO:tasks.rados.rados.0.smithi159.stdout:1876: left oid 45 (ObjNum 358 snap 52 seq_num 358) 2024-09-13T13:52:16.209 INFO:tasks.rados.rados.0.smithi159.stdout:1878: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:16.217 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1255766 2024-09-13T13:52:16.264 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:16.263+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:16.268 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:16.268 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:16.269 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.269 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:16.269 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:16.269 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:16.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:16.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:16.293 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:16.343 INFO:tasks.rados.rados.0.smithi159.stdout:1880: got (22) Invalid argument 2024-09-13T13:52:16.344 INFO:tasks.rados.rados.0.smithi159.stdout:1879: finishing set_chunk to oid 15 2024-09-13T13:52:16.344 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 631808 length: 590336 2024-09-13T13:52:16.344 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 0 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-09-13T13:52:16.344 INFO:tasks.rados.rados.0.smithi159.stdout:1877: expect (ObjNum 336 snap 47 seq_num 336) 2024-09-13T13:52:16.365 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:16.364+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:16.381 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:16.382 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:16.382 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.390 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:16.394 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1691568 2024-09-13T13:52:16.415 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1873: done (7 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1874: done (6 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1875: done (5 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1876: done (4 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1877: done (3 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1878: done (2 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1879: done (1 left) 2024-09-13T13:52:16.458 INFO:tasks.rados.rados.0.smithi159.stdout:1880: done (0 left) 2024-09-13T13:52:16.459 INFO:tasks.rados.rados.0.smithi159.stdout:1881: write oid 48 current snap is 52 2024-09-13T13:52:16.459 INFO:tasks.rados.rados.0.smithi159.stdout:1881: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-13T13:52:16.462 INFO:tasks.rados.rados.0.smithi159.stdout:1881: writing smithi15938319-48 from 433713 to 916695 tid 1 2024-09-13T13:52:16.466 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:16.465+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:16.469 INFO:tasks.rados.rados.0.smithi159.stdout:1881: writing smithi15938319-48 from 1424565 to 2033713 tid 2 2024-09-13T13:52:16.469 INFO:tasks.rados.rados.0.smithi159.stdout:1882: read oid 15 snap -1 2024-09-13T13:52:16.469 INFO:tasks.rados.rados.0.smithi159.stdout:1882: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:52:16.469 INFO:tasks.rados.rados.0.smithi159.stdout:1883: write (excl) oid 23 current snap is 52 2024-09-13T13:52:16.469 INFO:tasks.rados.rados.0.smithi159.stdout:1883: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-13T13:52:16.474 INFO:tasks.rados.rados.0.smithi159.stdout:1883: writing smithi15938319-23 from 450520 to 1008751 tid 1 2024-09-13T13:52:16.480 INFO:tasks.rados.rados.0.smithi159.stdout:1883: writing smithi15938319-23 from 1483047 to 1998172 tid 2 2024-09-13T13:52:16.480 INFO:tasks.rados.rados.0.smithi159.stdout:1884: snap_remove snap 35 2024-09-13T13:52:16.500 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 124.237521266, 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.501 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:52:16.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 124.236567767, 2024-09-13T13:52:16.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:52:16.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:52:16.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 112.72573150700001, 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.512 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:52:16.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:52:16.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:52:16.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.172820934000001, 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:52:16.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.172072044999993, 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:52:16.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.753737862999998, 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:16.534 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.748609505000005, 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:52:16.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:52:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:16.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:52:16.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:16.540 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.747697948999999, 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:16.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:52:16.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:52:16.545 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.719611861000004, 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:16.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.550 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2559 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'82, mlcod=84'79)", 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 74.653711565999998, 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040899279999999996, 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845545+0000", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845546+0000", 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:16.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845704+0000", 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015839000000000001 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845732+0000", 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7736e-05 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845738+0000", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6230000000000002e-06 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845791+0000", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3609999999999997e-05 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.845807+0000", 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5755000000000001e-05 2024-09-13T13:52:16.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849592+0000", 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0037853499999999998 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849635+0000", 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2669000000000001e-05 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:16.556 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 74.650337516999997, 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:52:16.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 74.649569468999999, 2024-09-13T13:52:16.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:16.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:52:16.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:16.566 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 69.293256189000004, 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3424 4.8 e118/34 4:1b24d731:::smithi15938319-13:head v 118'122, mlcod=115'119)", 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.516648965000002, 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041392160000000002, 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.34e-07 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983037+0000", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042880899999999999 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983075+0000", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7994000000000002e-05 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983083+0000", 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9030000000000004e-06 2024-09-13T13:52:16.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983136+0000", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3353999999999998e-05 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983152+0000", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5583999999999999e-05 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986717+0000", 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035656059999999998 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986747+0000", 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9532000000000001e-05 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.085848923999997, 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.193163945000002, 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:16.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:52:16.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.541572294000002, 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4198 4.8 e144/34 4:165e03eb:::smithi15938319-32:head v 144'170, mlcod=144'168)", 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 9.3878006890000005, 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041002770000000003, 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111457+0000", 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.597 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111960+0000", 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050279399999999998 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112261+0000", 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00030132899999999997 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112267+0000", 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.139e-06 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112323+0000", 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5773999999999998e-05 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112346+0000", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3425999999999999e-05 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.115519+0000", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031729390000000001 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.115556+0000", 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6854999999999999e-05 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:256 4.8 e150/34 4:165e03eb:::smithi15938319-32:2c v 150'186, mlcod=148'183)", 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.742261257, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045821789999999996, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756996+0000", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.98e-07 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757009+0000", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2948e-05 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757049+0000", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0793000000000001e-05 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757058+0000", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8380000000000008e-06 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757115+0000", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6483000000000001e-05 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757139+0000", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4167000000000001e-05 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761524+0000", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004384888 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761578+0000", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3863999999999998e-05 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:257 4.8 e150/34 4:1b24d731:::smithi15938319-13:2c v 150'188, mlcod=148'183)", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 3.7412205699999999, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063443070000000004, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758037+0000", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9999999999999996e-07 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758044+0000", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.759e-06 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758116+0000", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1913000000000001e-05 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758123+0000", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.763e-06 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758187+0000", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3283999999999999e-05 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758211+0000", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4728000000000001e-05 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764325+0000", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006114113 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764380+0000", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4947000000000001e-05 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:16.612 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:16.611+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:16.713 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:16.712+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:16.755 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.755 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:16.755 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:16.755 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1884: done (3 left) 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1885: delete oid 13 current snap is 52 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1881: finishing write tid 1 to smithi15938319-48 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1881: finishing write tid 2 to smithi15938319-48 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1881: finishing write tid 3 to smithi15938319-48 2024-09-13T13:52:16.758 INFO:tasks.rados.rados.0.smithi159.stdout:1881: finishing write tid 4 to smithi15938319-48 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:1881: oid 48 updating version 0 to 419 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:1881: oid 48 updating version 419 to 420 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:1881: oid 48 version 420 is already newer than 418 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 420 (ObjNum 359 snap 52 seq_num 359) dirty exists 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:1881: left oid 48 (ObjNum 359 snap 52 seq_num 359) 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:1882: expect (ObjNum 160 snap 13 seq_num 160) 2024-09-13T13:52:16.759 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:16.759 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:16.758+0000 7f26bcfe2640 -1 osd.3 153 handle_osd_map injecting map crc failure 2024-09-13T13:52:16.763 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:16.761+0000 7f2282f0c640 -1 osd.10 153 handle_osd_map injecting map crc failure 2024-09-13T13:52:16.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:16.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:52:16.813 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:16.812+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:16.869 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:16.870 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:16.870 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:16.870 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:16.870 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:16.878 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:16.908 INFO:tasks.rados.rados.0.smithi159.stdout:1883: finishing write tid 1 to smithi15938319-23 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: finishing write tid 2 to smithi15938319-23 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: finishing write tid 3 to smithi15938319-23 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: finishing write tid 4 to smithi15938319-23 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: oid 23 updating version 0 to 301 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: oid 23 updating version 301 to 302 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: oid 23 version 302 is already newer than 300 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 23 v 302 (ObjNum 360 snap 52 seq_num 360) dirty exists 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: left oid 23 (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1881: done (3 left) 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1882: done (2 left) 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1883: done (1 left) 2024-09-13T13:52:16.909 INFO:tasks.rados.rados.0.smithi159.stdout:1885: done (0 left) 2024-09-13T13:52:16.910 INFO:tasks.rados.rados.0.smithi159.stdout:1886: write (excl) oid 25 current snap is 52 2024-09-13T13:52:16.910 INFO:tasks.rados.rados.0.smithi159.stdout:1886: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-13T13:52:16.914 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:16.913+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:16.915 INFO:tasks.rados.rados.0.smithi159.stdout:1886: writing smithi15938319-25 from 484134 to 1192863 tid 1 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1886: writing smithi15938319-25 from 1883658 to 2084134 tid 2 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1887: tier_flush oid 13 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1887: tier_flush oid 13 snap -1 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1888: tier_promote oid 20 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1889: delete oid 26 current snap is 52 2024-09-13T13:52:16.920 INFO:tasks.rados.rados.0.smithi159.stdout:1887: got (16) Device or resource busy 2024-09-13T13:52:16.921 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 13 v 0 (ObjNum 273 snap 0 seq_num 0) unflushed 52 2024-09-13T13:52:16.929 INFO:tasks.rados.rados.0.smithi159.stdout:1886: finishing write tid 1 to smithi15938319-25 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:1888: got (0) Success 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 20 v 111 (ObjNum 271 snap 36 seq_num 271) dirty exists 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:1887: done (3 left) 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:1888: done (2 left) 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:1889: done (1 left) 2024-09-13T13:52:16.933 INFO:tasks.rados.rados.0.smithi159.stdout:1890: snap_create 2024-09-13T13:52:16.934 INFO:tasks.rados.rados.0.smithi159.stdout:1886: finishing write tid 2 to smithi15938319-25 2024-09-13T13:52:16.935 INFO:tasks.rados.rados.0.smithi159.stdout:1886: finishing write tid 3 to smithi15938319-25 2024-09-13T13:52:16.935 INFO:tasks.rados.rados.0.smithi159.stdout:1886: finishing write tid 4 to smithi15938319-25 2024-09-13T13:52:16.935 INFO:tasks.rados.rados.0.smithi159.stdout:1886: oid 25 updating version 0 to 253 2024-09-13T13:52:16.936 INFO:tasks.rados.rados.0.smithi159.stdout:1886: oid 25 version 253 is already newer than 252 2024-09-13T13:52:16.936 INFO:tasks.rados.rados.0.smithi159.stdout:1886: oid 25 version 253 is already newer than 251 2024-09-13T13:52:16.936 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 253 (ObjNum 362 snap 52 seq_num 362) dirty exists 2024-09-13T13:52:16.936 INFO:tasks.rados.rados.0.smithi159.stdout:1886: left oid 25 (ObjNum 362 snap 52 seq_num 362) 2024-09-13T13:52:17.015 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:17.015 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:17.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.471349476, 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:52:17.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:52:17.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.81190741, 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.021 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:52:17.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 146.81051918599999, 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:17.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:52:17.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:52:17.030 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 124.09653388700001, 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.031 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:52:17.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:52:17.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:17.037 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 123.24269513900001, 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:52:17.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:52:17.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:52:17.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.043 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.82326027000001, 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:52:17.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.82321232, 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.049 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:52:17.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.13474939300001, 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:52:17.054 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:17.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:52:17.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.060 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 91.439142266000005, 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.061 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:52:17.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:52:17.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.065 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:52:17.066 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.711439991999995, 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.067 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:52:17.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:52:17.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.685264849999996, 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.072 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:52:17.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 89.685223833999999, 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:52:17.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.078 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.079 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.080 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:17.081 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 64.866086925999994, 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.082 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.083 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:52:17.084 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:52:17.085 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.086 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.087 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3275 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.071107832999999, 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061117630000000001, 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3275 2024-09-13T13:52:17.088 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:17.089 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942258+0000", 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9369999999999998e-06 2024-09-13T13:52:17.092 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942265+0000", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-06 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942282+0000", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6880000000000001e-05 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942292+0000", 2024-09-13T13:52:17.093 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8689999999999999e-06 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7336000000000005e-05 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942388+0000", 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3090000000000001e-06 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.094 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945082+0000", 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026939160000000002 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945163+0000", 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0728000000000004e-05 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945384+0000", 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221375 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.095 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948177+0000", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027923119999999999 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948231+0000", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948299+0000", 2024-09-13T13:52:17.096 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8300000000000007e-05 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948368+0000", 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8872000000000001e-05 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.097 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3276 4.9 4:977d569f:::smithi15938319-22:head [rollback 12] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.071043348000003, 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082025069999999995, 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3276 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.098 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.099 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942325+0000", 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.512e-06 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942335+0000", 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0016e-05 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.100 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942341+0000", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942471+0000", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013015300000000001 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942478+0000", 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.55e-06 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.101 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946541+0000", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040631779999999998 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946640+0000", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8694000000000001e-05 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946923+0000", 2024-09-13T13:52:17.102 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028351899999999998 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948388+0000", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001464414 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950419+0000", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020313689999999999 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950465+0000", 2024-09-13T13:52:17.103 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5519999999999998e-05 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950523+0000", 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.842e-05 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3277 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:17.104 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.070985026999999, 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010801006, 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3277 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.105 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942380+0000", 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.106 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942384+0000", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1949999999999996e-06 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942391+0000", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0339999999999999e-06 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942396+0000", 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8409999999999999e-06 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.107 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942508+0000", 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112458 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942512+0000", 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.579e-06 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.108 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947640+0000", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051287859999999998 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947737+0000", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6329999999999994e-05 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947934+0000", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019687900000000001 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.109 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948478+0000", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000544448 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953066+0000", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00458831 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953129+0000", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2280999999999995e-05 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.110 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953180+0000", 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3520 4.9 4:977d569f:::smithi15938319-22:head [read 627200~1132544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.038781896000003, 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062848280000000001, 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.111 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3520 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:17.112 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974583+0000", 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974589+0000", 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4879999999999998e-06 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.113 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974600+0000", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1232e-05 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974609+0000", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3430000000000002e-06 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974667+0000", 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7388e-05 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.114 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974672+0000", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6930000000000004e-06 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979695+0000", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050231449999999997 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979758+0000", 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2232999999999999e-05 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.115 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.980867+0000", 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001109229 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3660 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 2c=[2c,2b,2a,29,28,27,26,25,24,23,22,1e,1a,14,12] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.310039589999999, 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068015810000000001, 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.116 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3660 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.117 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703325+0000", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.91e-07 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703326+0000", 2024-09-13T13:52:17.118 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6759999999999999e-06 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703331+0000", 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6879999999999998e-06 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703335+0000", 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8940000000000003e-06 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.119 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706175+0000", 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028405430000000001 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706229+0000", 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3813000000000001e-05 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706343+0000", 2024-09-13T13:52:17.120 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114123 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706835+0000", 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049211600000000002 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710024+0000", 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003188116 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.121 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710080+0000", 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6168999999999997e-05 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710126+0000", 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6152000000000001e-05 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.122 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4099 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1749291] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.932931973000001, 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059876800000000004, 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4099 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.123 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8599999999999998e-07 2024-09-13T13:52:17.124 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080436+0000", 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.309e-06 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080443+0000", 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6920000000000002e-06 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.125 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080449+0000", 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6980000000000003e-06 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080533+0000", 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3633000000000004e-05 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080540+0000", 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7830000000000001e-06 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.126 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086023+0000", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054832650000000002 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086131+0000", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107798 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086420+0000", 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028891600000000001 2024-09-13T13:52:17.127 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4100 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1749291] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 13.932904579000001, 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0066065960000000002, 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:17.128 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4100 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080460+0000", 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6199999999999998e-07 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080462+0000", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1560000000000002e-06 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080468+0000", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6380000000000001e-06 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080474+0000", 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8920000000000003e-06 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080576+0000", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000101439 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080582+0000", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5769999999999999e-06 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086462+0000", 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058799810000000003 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086579+0000", 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000116864 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.087066+0000", 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048658700000000002 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:17.133 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:17.134 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:17.013+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:17.135 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:17.134+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:17.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:17.236 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:17.235+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:17.242 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:17.242 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:17.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:17.337 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:17.336+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:17.359 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.359 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:17.360 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:17.360 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:17.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:17.438 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:17.437+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:17.477 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 113.122349888, 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:17.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:52:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 112.48289770700001, 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:52:17.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:52:17.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 112.48030075, 2024-09-13T13:52:17.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:17.491 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.74059100700001, 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:52:17.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.72714628200001, 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:52:17.495 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.832199568999997, 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.499 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:52:17.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:52:17.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.788484484999998, 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:52:17.503 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.594670902000004, 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:17.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:52:17.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:52:17.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 85.714279059000006, 2024-09-13T13:52:17.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:52:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.499187714000001, 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.517 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.521 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.498309285000005, 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:52:17.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:52:17.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.526 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 82.474593787000003, 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.527 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:17.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:52:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:52:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.701231738000004, 2024-09-13T13:52:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:52:17.533 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.537 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.538 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:17.539 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:52:17.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:52:17.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.700630849000007, 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:52:17.544 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:52:17.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.693478271999993, 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.549 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:52:17.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.553 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.387713857999998, 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:17.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:52:17.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.380082325999993, 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:17.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:52:17.561 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:52:17.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.668611182999996, 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.564 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:52:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:52:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:52:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:52:17.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:52:17.572 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.667676216000004, 2024-09-13T13:52:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:52:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.573 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:52:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:52:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:52:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:52:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:52:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:52:17.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:52:17.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:52:17.583 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.656993381999996, 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.584 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:52:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:52:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:52:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:52:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:52:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:52:17.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.594 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.594 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:17.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:17.596 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:17.594+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:17.696 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:17.695+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:17.703 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:17.703 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:17.703 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:17.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:17.756 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:52:17.759 INFO:tasks.rados.rados.0.smithi159.stdout:1886: done (1 left) 2024-09-13T13:52:17.760 INFO:tasks.rados.rados.0.smithi159.stdout:1890: done (0 left) 2024-09-13T13:52:17.760 INFO:tasks.rados.rados.0.smithi159.stdout:1891: snap_create 2024-09-13T13:52:17.796 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:17.796+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:17.821 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:17.821 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:17.821 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:17.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:17.897 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:17.897+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.939 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.771737231, 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.940 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:52:17.941 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.942 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.943 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.944 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.945 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.77171123799999, 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.946 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.947 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:52:17.948 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.949 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:52:17.950 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.666402416, 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:52:17.951 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:52:17.953 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.954 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.955 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.956 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.370367239, 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.957 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.958 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:52:17.959 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.960 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:52:17.961 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.962 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.370352739, 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.963 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.964 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.965 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.966 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:52:17.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.36809929899999, 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.968 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.969 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:52:17.970 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.971 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.972 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.245676011, 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.973 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.974 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:52:17.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:52:17.977 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.245645192, 2024-09-13T13:52:17.978 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:17.979 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.980 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.981 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:52:17.982 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.983 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.24348708399999, 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.984 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.985 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.986 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:52:17.987 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:17.988 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.03395288599999, 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.989 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.990 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.16797535400001, 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:52:17.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.994 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.995 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:52:17.996 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.997 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:52:17.998 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 124.15429964, 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:17.999 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.000 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:52:18.002 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.003 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.138398056, 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:52:18.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.005 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.006 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:52:18.007 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.008 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:52:18.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.010 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.011 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.130106561000005, 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.012 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.014 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.016 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.017 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:52:18.018 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.019 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.12607964, 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:52:18.020 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.021 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:52:18.022 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.023 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:52:18.024 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.025 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.026 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:52:18.027 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.125754684, 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.028 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.029 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:52:18.030 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:52:18.032 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.033 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.034 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:52:18.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 76.125723159000003, 2024-09-13T13:52:18.036 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.038 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.039 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:52:18.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:18.041 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 73.098033158000007, 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:52:18.043 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.045 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3255 4.a 4:5004af91:::smithi15938319-34:head [append 0~0] snapc 23=[23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.189450301000001, 2024-09-13T13:52:18.047 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.285444431, 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3255 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:18.048 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.049 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747900+0000", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1220000000000002e-06 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747906+0000", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5979999999999996e-06 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747911+0000", 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4e-06 2024-09-13T13:52:18.050 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750840+0000", 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029289250000000002 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750936+0000", 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5449e-05 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.051 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751081+0000", 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014517800000000001 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751499+0000", 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00041844799999999998 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033276+0000", 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28177714300000001 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.052 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033312+0000", 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5370999999999997e-05 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033342+0000", 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0457999999999999e-05 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.053 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:3255 4.a e112/85 ondisk, result = 0)", 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 49.184811207000003, 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.280808787, 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.054 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752540+0000", 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9849999999999998e-06 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033199+0000", 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28065851600000002 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033203+0000", 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5029999999999999e-06 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.056 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033267+0000", 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3596999999999994e-05 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033273+0000", 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.789e-06 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.057 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033346+0000", 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2960999999999995e-05 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.058 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:18.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:18.060 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:18.058+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:18.161 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:18.160+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:18.167 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:18.167 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:18.167 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:18.176 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:18.262 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:18.261+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:18.273 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:18.283 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:18.283 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:18.283 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:18.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:18.348 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:18.346+0000 7ffaeece8640 -1 osd.0 153 handle_osd_map injecting map crc failure 2024-09-13T13:52:18.348 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:18.347+0000 7fc16a180640 -1 osd.11 154 handle_osd_map injecting map crc failure 2024-09-13T13:52:18.362 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:18.361+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:18.411 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:18.411 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:18.435 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:18.446 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.545524254, 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.447 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:52:18.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.449 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 125.545456397, 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.453 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:52:18.455 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.456 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.21548178499999, 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.459 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:52:18.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.21544731, 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.464 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:18.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.466 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:52:18.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.469 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.21543232400001, 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:52:18.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:52:18.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.474 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.215418275, 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:18.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:52:18.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:18.479 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.21539656, 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:18.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.21536471500001, 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:52:18.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.053667989, 2024-09-13T13:52:18.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:18.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:52:18.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.493 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:52:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.053634839, 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.495 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:18.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:52:18.562 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:52:18.563 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:52:18.564 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:52:18.565 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.05360462100001, 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:52:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.05356227199999, 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:52:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:52:18.577 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:18.578 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.053528055, 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:18.579 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:52:18.581 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.582 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:52:18.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 110.05349478399999, 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.585 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:18.586 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.587 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:52:18.588 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.589 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:52:18.590 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:52:18.591 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 109.651211551, 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.593 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.594 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:52:18.595 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 75.626144941999996, 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.596 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:52:18.597 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.598 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.599 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:52:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2965 4.c 4:358f784f:::smithi15938319-41:head [append 0~0] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 61.570066419, 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014521253, 2024-09-13T13:52:18.602 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2965 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.603 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873612+0000", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2999999999999999e-07 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873613+0000", 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.398e-06 2024-09-13T13:52:18.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873616+0000", 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7190000000000001e-06 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873618+0000", 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6170000000000001e-06 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.605 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873745+0000", 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126389 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873747+0000", 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4030000000000001e-06 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882806+0000", 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090587889999999994 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882861+0000", 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4793999999999997e-05 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882955+0000", 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4377999999999997e-05 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.607 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883725+0000", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076983499999999996 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888075+0000", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043496630000000001 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888105+0000", 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0735000000000003e-05 2024-09-13T13:52:18.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888133+0000", 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7302999999999999e-05 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:4008 4.c e138/52 ondisk, result = 0)", 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:18.609 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 18.353895203, 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95524980500000001, 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.610 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2799999999999996e-07 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089787+0000", 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.422e-06 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044661+0000", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95487436400000003 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044676+0000", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4746000000000001e-05 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044804+0000", 2024-09-13T13:52:18.612 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012786699999999999 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044823+0000", 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9296e-05 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.045033+0000", 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00020948200000000001 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4104 4.c 4:358f784f:::smithi15938319-41:head [write 463959~478445,stat] snapc 30=[30,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 15.357858821000001, 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013243293, 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.614 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4104 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.615 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085821+0000", 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.199e-06 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086649+0000", 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082724800000000005 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.616 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086672+0000", 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3768e-05 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086684+0000", 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1584e-05 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086762+0000", 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8566000000000006e-05 2024-09-13T13:52:18.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086770+0000", 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091302+0000", 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045321670000000001 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.618 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091427+0000", 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012461500000000001 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091571+0000", 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014418699999999999 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098930+0000", 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0073584499999999999 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098994+0000", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4639000000000002e-05 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099030+0000", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5627000000000003e-05 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099062+0000", 2024-09-13T13:52:18.620 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4172 4.c 4:34327e96:::smithi15938319-44:head [write 581608~721541] snapc 30=[30,2d,2a,28,25,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 12.349340871000001, 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012273429000000001, 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:18.621 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4172 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.622 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094337+0000", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.094339+0000", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0930000000000001e-06 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095120+0000", 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000781268 2024-09-13T13:52:18.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095139+0000", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8553999999999999e-05 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095147+0000", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2700000000000004e-06 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095231+0000", 2024-09-13T13:52:18.624 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3923000000000003e-05 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.095236+0000", 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7589999999999997e-06 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099704+0000", 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044682710000000002 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099783+0000", 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.9190000000000006e-05 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.099874+0000", 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0242999999999996e-05 2024-09-13T13:52:18.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.101994+0000", 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021203110000000002 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106540+0000", 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004545922 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.627 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106578+0000", 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8114999999999997e-05 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:06.106610+0000", 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2509999999999999e-05 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:18.628 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:18.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:18.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:18.631 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:18.631 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:18.632 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:18.632 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:18.632 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:18.632 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:18.633 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:18.633 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:18.633 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:18.633 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:18.634 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:18.634 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:18.634 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:18.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:18.635 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:18.635 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:18.635 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:18.636 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:18.636 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:18.629+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:18.650 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:18.651 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:18.673 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:18.673 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:18.695 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:18.695 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:18.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:18.699+0000 7f44c67fc640 0 -- 172.21.15.27:0/3952399966 >> v2:172.21.15.159:6805/4073091144 conn(0x7f44c82222e0 msgr2=0x7f44c82246d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:52:18.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:18.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:18.733 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:18.732+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:18.749 INFO:tasks.rados.rados.0.smithi159.stdout:1891: done (0 left) 2024-09-13T13:52:18.749 INFO:tasks.rados.rados.0.smithi159.stdout:1892: set_chunk oid 8 target oid 2024-09-13T13:52:18.753 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:18.753 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1892: set_chunk oid 8 offset: 316416 length: 241152 target oid b7050b6b6aae71f090e938b320864059d8228306c3b7a1414199fdb11633bd96 offset: 0 snap -1 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1893: read oid 39 snap 53 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1893: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1894: read oid 14 snap -1 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1894: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:18.759 INFO:tasks.rados.rados.0.smithi159.stdout:1895: set_chunk oid 18 target oid 2024-09-13T13:52:18.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:18.847 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:18.846+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:18.906 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:18.907 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:18.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:18.956 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:18.956+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:19.036 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:19.034+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:19.058 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.058 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:19.137 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:19.136+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:19.210 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.210 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.212 INFO:tasks.rados.rados.0.smithi159.stdout:1895: set_chunk oid 18 offset: 758272 length: 1495040 target oid d4eb3305d6755e5e9a82c17065b361e95da0bdbb9aa71454e7596c6ba998740e offset: 0 snap -1 2024-09-13T13:52:19.212 INFO:tasks.rados.rados.0.smithi159.stdout:1896: delete oid 47 current snap is 54 2024-09-13T13:52:19.212 INFO:tasks.rados.rados.0.smithi159.stdout:1892: finishing set_chunk to oid 8 2024-09-13T13:52:19.213 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 212 (ObjNum 352 snap 50 seq_num 352) dirty exists 2024-09-13T13:52:19.213 INFO:tasks.rados.rados.0.smithi159.stdout:1893: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:19.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:19.238 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:19.237+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:19.320 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.320 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.328 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:19.338 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:19.337+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:19.349 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:19.417 INFO:tasks.rados.rados.0.smithi159.stdout:1894: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:19.431 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.431 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.439 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:19.438+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:19.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:19.540 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:19.539+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:19.584 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.584 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.588 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2216923 2024-09-13T13:52:19.593 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1895: finishing set_chunk to oid 18 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 18 v 219 (ObjNum 266 snap 32 seq_num 266) dirty exists 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1892: done (4 left) 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1893: done (3 left) 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1894: done (2 left) 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1895: done (1 left) 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1896: done (0 left) 2024-09-13T13:52:19.614 INFO:tasks.rados.rados.0.smithi159.stdout:1897: write (excl) oid 41 current snap is 54 2024-09-13T13:52:19.615 INFO:tasks.rados.rados.0.smithi159.stdout:1897: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-13T13:52:19.620 INFO:tasks.rados.rados.0.smithi159.stdout:1897: writing smithi15938319-41 from 534555 to 1185384 tid 1 2024-09-13T13:52:19.625 INFO:tasks.rados.rados.0.smithi159.stdout:1897: writing smithi15938319-41 from 1659104 to 2134555 tid 2 2024-09-13T13:52:19.625 INFO:tasks.rados.rados.0.smithi159.stdout:1898: read oid 22 snap -1 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1898: expect deleted 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1899: read oid 17 snap -1 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1899: expect (ObjNum 348 snap 49 seq_num 348) 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1900: read oid 23 snap -1 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1900: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1901: read oid 26 snap -1 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1901: expect deleted 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1902: write oid 29 current snap is 54 2024-09-13T13:52:19.626 INFO:tasks.rados.rados.0.smithi159.stdout:1902: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-13T13:52:19.633 INFO:tasks.rados.rados.0.smithi159.stdout:1902: writing smithi15938319-29 from 551362 to 1277440 tid 1 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1902: writing smithi15938319-29 from 1717586 to 2127365 tid 2 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1903: read oid 1 snap 52 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1903: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: finishing write tid 1 to smithi15938319-41 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1901: done (6 left) 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1904: delete oid 36 current snap is 54 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: finishing write tid 2 to smithi15938319-41 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: finishing write tid 3 to smithi15938319-41 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: finishing write tid 4 to smithi15938319-41 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: oid 41 updating version 0 to 334 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: oid 41 updating version 334 to 335 2024-09-13T13:52:19.639 INFO:tasks.rados.rados.0.smithi159.stdout:1897: oid 41 updating version 335 to 336 2024-09-13T13:52:19.640 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 336 (ObjNum 365 snap 54 seq_num 365) dirty exists 2024-09-13T13:52:19.640 INFO:tasks.rados.rados.0.smithi159.stdout:1897: left oid 41 (ObjNum 365 snap 54 seq_num 365) 2024-09-13T13:52:19.640 INFO:tasks.rados.rados.0.smithi159.stdout:1900: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:19.640 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:19.641 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:19.639+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:19.683 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:19.728 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:19.738 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.738 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.742 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:19.741+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:19.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:19.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:19.773 INFO:tasks.rados.rados.0.smithi159.stdout:1899: expect (ObjNum 348 snap 49 seq_num 348) 2024-09-13T13:52:19.832 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2889271 2024-09-13T13:52:19.843 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:19.842+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:19.850 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.851 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:19.917 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3624700 2024-09-13T13:52:19.919 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:19.944 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:19.943+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:19.962 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:19.963 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:19.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:20.009 INFO:tasks.rados.rados.0.smithi159.stdout:1902: finishing write tid 1 to smithi15938319-29 2024-09-13T13:52:20.009 INFO:tasks.rados.rados.0.smithi159.stdout:1902: finishing write tid 2 to smithi15938319-29 2024-09-13T13:52:20.009 INFO:tasks.rados.rados.0.smithi159.stdout:1902: finishing write tid 3 to smithi15938319-29 2024-09-13T13:52:20.009 INFO:tasks.rados.rados.0.smithi159.stdout:1902: finishing write tid 4 to smithi15938319-29 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:1902: oid 29 updating version 0 to 433 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:1902: oid 29 version 433 is already newer than 432 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:1902: oid 29 updating version 433 to 434 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 29 v 434 (ObjNum 366 snap 54 seq_num 366) dirty exists 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:1902: left oid 29 (ObjNum 366 snap 54 seq_num 366) 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:1903: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:20.010 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:20.046 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:20.045+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:20.074 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.074 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.076 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:20.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:20.138 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:20.147 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:20.145+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:20.185 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.185 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1897: done (6 left) 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1898: done (5 left) 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1899: done (4 left) 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1900: done (3 left) 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1902: done (2 left) 2024-09-13T13:52:20.204 INFO:tasks.rados.rados.0.smithi159.stdout:1903: done (1 left) 2024-09-13T13:52:20.205 INFO:tasks.rados.rados.0.smithi159.stdout:1904: done (0 left) 2024-09-13T13:52:20.205 INFO:tasks.rados.rados.0.smithi159.stdout:1905: read oid 50 snap -1 2024-09-13T13:52:20.205 INFO:tasks.rados.rados.0.smithi159.stdout:1905: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:20.205 INFO:tasks.rados.rados.0.smithi159.stdout:1906: delete oid 13 current snap is 54 2024-09-13T13:52:20.205 INFO:tasks.rados.rados.0.smithi159.stdout:1906: done (1 left) 2024-09-13T13:52:20.206 INFO:tasks.rados.rados.0.smithi159.stdout:1907: read oid 10 snap 50 2024-09-13T13:52:20.206 INFO:tasks.rados.rados.0.smithi159.stdout:1907: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:20.206 INFO:tasks.rados.rados.0.smithi159.stdout:1908: write (excl) oid 17 current snap is 54 2024-09-13T13:52:20.206 INFO:tasks.rados.rados.0.smithi159.stdout:1908: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-13T13:52:20.211 INFO:tasks.rados.rados.0.smithi159.stdout:1908: writing smithi15938319-17 from 601783 to 1153608 tid 1 2024-09-13T13:52:20.217 INFO:tasks.rados.rados.0.smithi159.stdout:1908: writing smithi15938319-17 from 1776679 to 2201783 tid 2 2024-09-13T13:52:20.217 INFO:tasks.rados.rados.0.smithi159.stdout:1909: set_chunk oid 6 target oid 2024-09-13T13:52:20.230 INFO:tasks.rados.rados.0.smithi159.stdout:1909: set_chunk oid 6 offset: 1799168 length: 278528 target oid 3aac7f4c1abbd1515e27905db601e0078a914ccc7e83f52b9a6f8cdb0238a1e1 offset: 0 snap -1 2024-09-13T13:52:20.230 INFO:tasks.rados.rados.0.smithi159.stdout:1910: tier_flush oid 42 2024-09-13T13:52:20.231 INFO:tasks.rados.rados.0.smithi159.stdout:1910: tier_flush oid 42 snap -1 2024-09-13T13:52:20.231 INFO:tasks.rados.rados.0.smithi159.stdout:1905: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:20.232 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:20.232+0000 7f002dbdb640 -1 osd.15 156 handle_osd_map injecting map crc failure 2024-09-13T13:52:20.247 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:20.247+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:20.297 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1710616 2024-09-13T13:52:20.310 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.310 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.319 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:20.348 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:20.347+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:20.388 INFO:tasks.rados.rados.0.smithi159.stdout:1907: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:20.421 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.421 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:20.448 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2824533 2024-09-13T13:52:20.474 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:20.473+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: finishing write tid 1 to smithi15938319-17 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: finishing write tid 2 to smithi15938319-17 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: finishing write tid 3 to smithi15938319-17 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: finishing write tid 4 to smithi15938319-17 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: oid 17 updating version 0 to 437 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: oid 17 version 437 is already newer than 436 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: oid 17 updating version 437 to 438 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 438 (ObjNum 369 snap 54 seq_num 369) dirty exists 2024-09-13T13:52:20.504 INFO:tasks.rados.rados.0.smithi159.stdout:1908: left oid 17 (ObjNum 369 snap 54 seq_num 369) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1905: done (4 left) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1907: done (3 left) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1908: done (2 left) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1911: read oid 26 snap 49 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1911: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1912: read oid 1 snap -1 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1912: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1913: write oid 47 current snap is 54 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1909: finishing set_chunk to oid 6 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 6 v 63 (ObjNum 253 snap 28 seq_num 253) dirty exists 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1910: got (16) Device or resource busy 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 42 v 185 (ObjNum 335 snap 47 seq_num 335) unflushed 47 2024-09-13T13:52:20.505 INFO:tasks.rados.rados.0.smithi159.stdout:1913: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-13T13:52:20.510 INFO:tasks.rados.rados.0.smithi159.stdout:1913: writing smithi15938319-47 from 618590 to 1245664 tid 1 2024-09-13T13:52:20.516 INFO:tasks.rados.rados.0.smithi159.stdout:1913: writing smithi15938319-47 from 1835161 to 2218590 tid 2 2024-09-13T13:52:20.516 INFO:tasks.rados.rados.0.smithi159.stdout:1909: done (4 left) 2024-09-13T13:52:20.516 INFO:tasks.rados.rados.0.smithi159.stdout:1910: done (3 left) 2024-09-13T13:52:20.516 INFO:tasks.rados.rados.0.smithi159.stdout:1914: tier_promote oid 30 2024-09-13T13:52:20.516 INFO:tasks.rados.rados.0.smithi159.stdout:1912: expect (ObjNum 338 snap 47 seq_num 338) 2024-09-13T13:52:20.534 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.534 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.543 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:20.550 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:20.549+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:20.586 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:20.585+0000 7f586a715640 -1 osd.9 156 handle_osd_map injecting map crc failure 2024-09-13T13:52:20.650 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:20.649+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:20.686 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.686 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.695 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:20.701 INFO:tasks.rados.rados.0.smithi159.stdout:1911: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:20.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:20.746 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:20.779 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:20.778+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:20.792 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:20.838 INFO:tasks.rados.rados.0.smithi159.stdout:1913: finishing write tid 1 to smithi15938319-47 2024-09-13T13:52:20.838 INFO:tasks.rados.rados.0.smithi159.stdout:1914: got (0) Success 2024-09-13T13:52:20.838 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 322 (ObjNum 347 snap 48 seq_num 347) dirty exists 2024-09-13T13:52:20.838 INFO:tasks.rados.rados.0.smithi159.stdout:1913: finishing write tid 2 to smithi15938319-47 2024-09-13T13:52:20.839 INFO:tasks.rados.rados.0.smithi159.stdout:1913: finishing write tid 3 to smithi15938319-47 2024-09-13T13:52:20.839 INFO:tasks.rados.rados.0.smithi159.stdout:1911: done (3 left) 2024-09-13T13:52:20.839 INFO:tasks.rados.rados.0.smithi159.stdout:1912: done (2 left) 2024-09-13T13:52:20.839 INFO:tasks.rados.rados.0.smithi159.stdout:1914: done (1 left) 2024-09-13T13:52:20.839 INFO:tasks.rados.rados.0.smithi159.stdout:1915: set_chunk oid 48 target oid 2024-09-13T13:52:20.839 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:20.839 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:20.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:20.861 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:20.861 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:20.884 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:20.884 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:20.888 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:20.887+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:20.905 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:20.905 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:20.926 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:20.927 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:20.927 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:20.949 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:20.949 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:20.952 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:20.951+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:20.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:20.993 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:20.993 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:20.994 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:20.994 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:20.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:20.994 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:20.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 152.44891454699999, 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.995 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:52:20.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:20.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:20.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:20.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:21.000 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.394588658, 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:52:21.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:52:21.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.38121026799999, 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.006 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:21.010 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 122.56787733100001, 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:52:21.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.762465324, 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.015 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:52:21.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.020 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.762428152, 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:52:21.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.026 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.762407424, 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.027 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:52:21.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:52:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.762311123, 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:52:21.033 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:52:21.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:52:21.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:52:21.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.76228847900001, 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.039 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:52:21.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:52:21.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.045 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.762262449, 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.046 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:52:21.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:52:21.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:52:21.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:52:21.054 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.055 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.055 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.055 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.056 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:21.056 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.622761582, 2024-09-13T13:52:21.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:52:21.056 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.056 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:52:21.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.058 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:21.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:52:21.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:52:21.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:52:21.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:52:21.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:52:21.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:52:21.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.070 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.070 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:21.070 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.622654635, 2024-09-13T13:52:21.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:21.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:52:21.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:21.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:52:21.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:52:21.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:52:21.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:52:21.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.622625376, 2024-09-13T13:52:21.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:21.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.087 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.616260973, 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.088 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:21.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:52:21.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:52:21.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:52:21.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:21.098 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.611300779, 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:21.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:52:21.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:52:21.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:52:21.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.66931106200001, 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.111 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:52:21.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:52:21.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:52:21.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.669271924, 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.118 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:52:21.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:52:21.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.127841927000006, 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:52:21.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:52:21.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.130 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.127778981000006, 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.131 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:21.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.138 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.127745298999997, 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:52:21.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:52:21.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.144 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.145 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:21.146 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:21.146 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:21.147 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:21.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:21.147 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:21.148 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:21.148 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:21.148 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:21.148 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:21.149 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:21.149 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:21.150 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:21.141+0000 7fca65b9c640 0 -- 172.21.15.27:0/430019115 >> v2:172.21.15.159:6805/4073091144 conn(0x7fca60222160 msgr2=0x7fca60224550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:52:21.150 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:21.144+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:21.164 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:21.164 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:21.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:21.247 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:21.246+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:21.253 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.253 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:21.253 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:21.253 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:21.347 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:21.347+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:21.364 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.365 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:21.365 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:21.365 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:21.365 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.374 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1915: set_chunk oid 48 offset: 154112 length: 1542656 target oid 92ffe9d3e6fcece08b9c54fc55e88dc4a847ca3a5ba716d467589f231d78541b offset: 0 snap -1 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1916: write oid 40 current snap is 54 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1913: finishing write tid 4 to smithi15938319-47 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1913: oid 47 updating version 0 to 288 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1913: oid 47 updating version 288 to 289 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1913: oid 47 updating version 289 to 290 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 290 (ObjNum 370 snap 54 seq_num 370) dirty exists 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1913: left oid 47 (ObjNum 370 snap 54 seq_num 370) 2024-09-13T13:52:21.427 INFO:tasks.rados.rados.0.smithi159.stdout:1916: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-13T13:52:21.433 INFO:tasks.rados.rados.0.smithi159.stdout:1916: writing smithi15938319-40 from 635397 to 1337720 tid 1 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1916: writing smithi15938319-40 from 1893643 to 2235397 tid 2 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1913: done (2 left) 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1917: read oid 35 snap 18 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1917: expect (ObjNum 198 snap 18 seq_num 198) 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1918: write oid 13 current snap is 54 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1915: finishing set_chunk to oid 48 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 420 (ObjNum 359 snap 52 seq_num 359) dirty exists 2024-09-13T13:52:21.439 INFO:tasks.rados.rados.0.smithi159.stdout:1918: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-13T13:52:21.446 INFO:tasks.rados.rados.0.smithi159.stdout:1918: writing smithi15938319-13 from 652204 to 1429776 tid 1 2024-09-13T13:52:21.448 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:21.448+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:21.451 INFO:tasks.rados.rados.0.smithi159.stdout:1918: writing smithi15938319-13 from 1952125 to 2252204 tid 2 2024-09-13T13:52:21.451 INFO:tasks.rados.rados.0.smithi159.stdout:1915: done (3 left) 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1919: snap_create 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: finishing write tid 1 to smithi15938319-40 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: finishing write tid 2 to smithi15938319-40 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: finishing write tid 3 to smithi15938319-40 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: finishing write tid 4 to smithi15938319-40 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: oid 40 updating version 0 to 284 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: oid 40 version 284 is already newer than 283 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: oid 40 updating version 284 to 285 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 285 (ObjNum 371 snap 54 seq_num 371) dirty exists 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1916: left oid 40 (ObjNum 371 snap 54 seq_num 371) 2024-09-13T13:52:21.452 INFO:tasks.rados.rados.0.smithi159.stdout:1917: expect (ObjNum 198 snap 18 seq_num 198) 2024-09-13T13:52:21.453 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:21.481 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.74237986, 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.482 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:52:21.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:52:21.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.737139238, 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:52:21.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:52:21.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.495 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.73708792400001, 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:52:21.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:52:21.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:52:21.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.737015727, 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.502 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:52:21.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.73697672200001, 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.508 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:52:21.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.73691231000001, 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.515 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:52:21.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:52:21.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.711562214, 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:52:21.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:52:21.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:52:21.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.527 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.68100139000001, 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.528 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:52:21.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:52:21.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.659674803, 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.534 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.538 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 128.58145852600001, 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:52:21.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:52:21.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:52:21.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.237486866, 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:52:21.545 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:52:21.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.237428814, 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.552 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:21.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:52:21.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.237402855, 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.237288439, 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.564 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.565 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:52:21.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:21.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.237257548, 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.575 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:52:21.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:52:21.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:52:21.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.23722096500001, 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:52:21.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:52:21.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.15742609500001, 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:52:21.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 99.782765677, 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.593 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:52:21.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.659574153999998, 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.597 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:21.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:52:21.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.651553856000007, 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:52:21.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:52:21.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.607 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.608 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:21.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:21.609 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:21.608+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: finishing write tid 1 to smithi15938319-13 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: finishing write tid 2 to smithi15938319-13 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: finishing write tid 3 to smithi15938319-13 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: finishing write tid 4 to smithi15938319-13 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: oid 13 updating version 0 to 196 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: oid 13 updating version 196 to 198 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: oid 13 version 198 is already newer than 197 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 198 (ObjNum 372 snap 54 seq_num 372) dirty exists 2024-09-13T13:52:21.648 INFO:tasks.rados.rados.0.smithi159.stdout:1918: left oid 13 (ObjNum 372 snap 54 seq_num 372) 2024-09-13T13:52:21.709 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:21.708+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:21.713 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.713 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:21.713 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:21.713 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.722 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:21.809 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:21.809+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:21.864 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:21.865 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:21.865 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:21.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:21.917 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:21.916+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:21.977 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 151.74761375400001, 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:52:21.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:52:21.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.242875572, 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:52:21.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:52:21.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:52:21.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.24282625999999, 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:52:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:21.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.997 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.24281055599999, 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.998 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:21.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:52:22.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.24278287999999, 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.004 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:52:22.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:52:22.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.010 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.20756528199999, 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:52:22.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:52:22.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.2004963, 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:52:22.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:52:22.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.165957838, 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.023 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:52:22.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:52:22.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.158327996, 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:52:22.029 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:22.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:52:22.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:52:22.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.035 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.15495162400001, 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.036 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:52:22.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:52:22.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.05209234399999, 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.042 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:22.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:52:22.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:52:22.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.048 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 124.841068804, 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.049 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:52:22.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 124.82955352099999, 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.054 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:52:22.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.72824505299999, 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:52:22.058 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:52:22.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:52:22.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.728180705, 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.064 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:52:22.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.728152643, 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:22.070 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:52:22.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:52:22.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.726470753, 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.076 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:52:22.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:52:22.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:52:22.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.286942545, 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:22.084 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:52:22.087 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.158306789999997, 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.088 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:52:22.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:52:22.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 51.013050190000001, 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.094 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:52:22.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:52:22.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:52:22.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:22.100 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:22.102 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:22.102 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:22.101+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:22.151 INFO:tasks.rados.rados.0.smithi159.stdout:1916: done (3 left) 2024-09-13T13:52:22.151 INFO:tasks.rados.rados.0.smithi159.stdout:1917: done (2 left) 2024-09-13T13:52:22.151 INFO:tasks.rados.rados.0.smithi159.stdout:1918: done (1 left) 2024-09-13T13:52:22.151 INFO:tasks.rados.rados.0.smithi159.stdout:1919: done (0 left) 2024-09-13T13:52:22.151 INFO:tasks.rados.rados.0.smithi159.stdout:1920: snap_create 2024-09-13T13:52:22.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:52:22.202 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:22.202+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:22.207 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.208 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:22.208 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:22.208 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:22.217 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:22.303 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:22.302+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:22.322 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.322 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:22.322 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:22.322 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:22.322 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:22.330 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:22.405 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:22.404+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.74249214899999, 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.477 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:52:22.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.483 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.74241474799999, 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.74226221000001, 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.489 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:52:22.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.742156171, 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.495 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.500 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.733817093, 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:52:22.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:52:22.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.706450383, 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.505 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:52:22.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.70468378499999, 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:52:22.511 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:52:22.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.699804443, 2024-09-13T13:52:22.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:52:22.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.68929271799999, 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:52:22.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:52:22.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.66503542800001, 2024-09-13T13:52:22.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:52:22.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:52:22.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.66215038300001, 2024-09-13T13:52:22.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:22.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:52:22.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:52:22.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.541 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.65994635800001, 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:52:22.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:52:22.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:52:22.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.652955714, 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:52:22.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 129.5479641, 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.554 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:52:22.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:52:22.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.558 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 91.724374238999999, 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:52:22.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 91.724341448999994, 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:22.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:52:22.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.569 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 91.608581942000001, 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:52:22.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 86.683979253000004, 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.574 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:52:22.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:52:22.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.580 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 58.733769506000002, 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.581 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:52:22.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3286 3.2 e114/73 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head v 114'133, mlcod=105'131)", 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 51.506800214000002, 2024-09-13T13:52:22.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74551292800000002, 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.73e-07 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.968632+0000", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075455699999999999 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711912+0000", 2024-09-13T13:52:22.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74328065200000004 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711933+0000", 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1279000000000001e-05 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712080+0000", 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014671199999999999 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.588 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712101+0000", 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0339999999999998e-05 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713357+0000", 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012564550000000001 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713390+0000", 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2759999999999998e-05 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.589 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.590 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.590 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.590 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:22.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:22.592 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:22.590+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:22.692 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:22.691+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:22.696 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.696 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:22.696 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:22.696 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:22.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:22.793 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:22.792+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:22.809 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:22.809 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:22.809 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:22.818 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:22.894 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:22.893+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.962 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.213530518, 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:52:22.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.21055390699999, 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.967 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:52:22.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:52:22.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.20857973599999, 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:22.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:52:22.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:52:22.976 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.16984921299999, 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:52:22.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:52:22.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.16594792199999, 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:52:22.981 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:52:22.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:52:22.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.04478293400001, 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.986 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:52:22.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 130.03998203899999, 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:52:22.992 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:52:22.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:22.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:52:22.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 128.885343202, 2024-09-13T13:52:22.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:22.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:52:23.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:52:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 114.70548171599999, 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:52:23.004 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:52:23.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:52:23.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:23.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.67239035999999, 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:52:23.010 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:52:23.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 87.163750105000005, 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.014 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:52:23.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.018 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 61.049722844999998, 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:52:23.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:52:23.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 60.052120486, 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:23.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:52:23.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:52:23.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4028 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.889578700000001, 2024-09-13T13:52:23.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010533003000000001, 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4028 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070849+0000", 2024-09-13T13:52:23.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070861+0000", 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1973999999999999e-05 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070881+0000", 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070894+0000", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2204999999999999e-05 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070965+0000", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1613999999999996e-05 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070971+0000", 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.395e-06 2024-09-13T13:52:23.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077011+0000", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006040035 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077114+0000", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103727 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081381+0000", 2024-09-13T13:52:23.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042663370000000003 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4029 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.889494656, 2024-09-13T13:52:23.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012297313000000001, 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4029 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:23.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7400000000000003e-07 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070935+0000", 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1089999999999998e-06 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070941+0000", 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070945+0000", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2980000000000002e-06 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071034+0000", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8661999999999999e-05 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071041+0000", 2024-09-13T13:52:23.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081442+0000", 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010401216 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081561+0000", 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119207 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083229+0000", 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001667948 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4030 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.889473541000001, 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722845, 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.047 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4030 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.048 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070955+0000", 2024-09-13T13:52:23.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5060000000000001e-06 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070958+0000", 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3160000000000001e-06 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070963+0000", 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8500000000000002e-06 2024-09-13T13:52:23.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071060+0000", 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7565999999999994e-05 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071063+0000", 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0479999999999999e-06 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083261+0000", 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012197846 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083377+0000", 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011555100000000001 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083676+0000", 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029883399999999999 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.053 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4357 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.207606891999999, 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012624672, 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.054 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4357 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752820+0000", 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1399999999999997e-07 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752827+0000", 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5010000000000001e-06 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752834+0000", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752838+0000", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752943+0000", 2024-09-13T13:52:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010495199999999999 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752947+0000", 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8789999999999996e-06 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762157+0000", 2024-09-13T13:52:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092093769999999995 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762260+0000", 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103145 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765444+0000", 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031843850000000001 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4358 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.207563247, 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013684981000000001, 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4358 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.060 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752864+0000", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4700000000000001e-07 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752867+0000", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4350000000000001e-06 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752874+0000", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3869999999999997e-06 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752880+0000", 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752994+0000", 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752999+0000", 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0270000000000003e-06 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765566+0000", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012566623000000001 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765652+0000", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6439000000000004e-05 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.766548+0000", 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000895753 2024-09-13T13:52:23.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4422 4.6 4:6ce02a65:::smithi15938319-20:head [tier-promote] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 6.0401903690000003, 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722221000000001, 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4422 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920238+0000", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.618e-06 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920244+0000", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3929999999999996e-06 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920260+0000", 2024-09-13T13:52:23.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6308e-05 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920270+0000", 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6490000000000003e-06 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920344+0000", 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3979000000000004e-05 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920351+0000", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9369999999999998e-06 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924882+0000", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045310389999999997 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924943+0000", 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0618000000000001e-05 2024-09-13T13:52:23.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.932958+0000", 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080156800000000007 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4423 4.6 4:6668b762:::smithi15938319-26:head [stat,delete] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 6.0401276929999996, 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012827221, 2024-09-13T13:52:23.070 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4423 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920302+0000", 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6089999999999999e-06 2024-09-13T13:52:23.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920309+0000", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.844e-06 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920314+0000", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.259e-06 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920407+0000", 2024-09-13T13:52:23.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3058000000000002e-05 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920414+0000", 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4159999999999998e-06 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925431+0000", 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005016553 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925484+0000", 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3507000000000001e-05 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925679+0000", 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019484 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.928735+0000", 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030555719999999999 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933007+0000", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042717700000000003 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933071+0000", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3935000000000004e-05 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933126+0000", 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5358000000000001e-05 2024-09-13T13:52:23.076 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.077 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:23.078 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:23.079 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:23.077+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:23.123 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:23.122+0000 7f2284f10640 -1 osd.10 157 handle_osd_map injecting map crc failure 2024-09-13T13:52:23.152 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:23.158 INFO:tasks.rados.rados.0.smithi159.stdout:1920: done (0 left) 2024-09-13T13:52:23.159 INFO:tasks.rados.rados.0.smithi159.stdout:1921: read oid 9 snap -1 2024-09-13T13:52:23.159 INFO:tasks.rados.rados.0.smithi159.stdout:1921: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:23.159 INFO:tasks.rados.rados.0.smithi159.stdout:1922: set_chunk oid 19 target oid 2024-09-13T13:52:23.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:23.178 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:23.178+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:23.184 INFO:tasks.rados.rados.0.smithi159.stdout:1922: set_chunk oid 19 offset: 866304 length: 1811456 target oid 35ba6d28bd110c1f52fc7debf85c437a389e41e6c49db39f068440db106e9eb2 offset: 0 snap 54 2024-09-13T13:52:23.184 INFO:tasks.rados.rados.0.smithi159.stdout:1923: rollback oid 49 current snap is 56 2024-09-13T13:52:23.184 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 49 to 32 2024-09-13T13:52:23.184 INFO:tasks.rados.rados.0.smithi159.stdout:1921: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:23.184 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:23.184 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:23.185 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:23.185 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:23.193 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:23.275 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:23.280 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:23.279+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:23.297 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:23.297 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:23.297 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:23.297 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:23.297 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:23.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1922: finishing set_chunk to oid 19 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1923: finishing rollback tid 1 to smithi15938319-49 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1923: finishing rollback tid 2 to smithi15938319-49 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 49 v 95 (ObjNum 173 snap 16 seq_num 173) dirty exists 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1921: done (2 left) 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1922: done (1 left) 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1923: done (0 left) 2024-09-13T13:52:23.328 INFO:tasks.rados.rados.0.smithi159.stdout:1924: delete oid 36 current snap is 56 2024-09-13T13:52:23.330 INFO:tasks.rados.rados.0.smithi159.stdout:1924: done (0 left) 2024-09-13T13:52:23.330 INFO:tasks.rados.rados.0.smithi159.stdout:1925: read oid 48 snap -1 2024-09-13T13:52:23.330 INFO:tasks.rados.rados.0.smithi159.stdout:1925: expect (ObjNum 359 snap 52 seq_num 359) 2024-09-13T13:52:23.330 INFO:tasks.rados.rados.0.smithi159.stdout:1926: delete oid 32 current snap is 56 2024-09-13T13:52:23.333 INFO:tasks.rados.rados.0.smithi159.stdout:1926: done (1 left) 2024-09-13T13:52:23.333 INFO:tasks.rados.rados.0.smithi159.stdout:1927: snap_create 2024-09-13T13:52:23.336 INFO:tasks.rados.rados.0.smithi159.stdout:1925: expect (ObjNum 359 snap 52 seq_num 359) 2024-09-13T13:52:23.345 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1679190 2024-09-13T13:52:23.348 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 81919 2024-09-13T13:52:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:23.350 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:23.373 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:23.373 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:23.380 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:23.379+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:23.395 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:23.395 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:23.412 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 131.149026123, 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.413 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.414 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:52:23.415 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.416 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:52:23.417 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 131.14807262400001, 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.418 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.419 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.420 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:52:23.421 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.422 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.423 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 119.637236364, 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:52:23.424 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.425 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.426 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:52:23.427 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.428 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 96.084325790999998, 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:52:23.429 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.430 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.431 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.432 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.433 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 96.083576902000004, 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.434 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:23.435 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.436 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:52:23.437 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.438 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 92.665242719999995, 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:52:23.439 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.440 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.441 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.444 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:52:23.445 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.446 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.447 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 92.660114362000002, 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.448 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:52:23.449 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 92.659202805999996, 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:23.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:52:23.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:137 4.8 e80/34 4:165e03eb:::smithi15938319-32:e v 80'73, mlcod=78'69)", 2024-09-13T13:52:23.457 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 86.631116718000001, 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041628480000000002, 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779645+0000", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3599999999999999e-07 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779652+0000", 2024-09-13T13:52:23.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2459999999999997e-06 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779708+0000", 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5452999999999999e-05 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779719+0000", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1663e-05 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779795+0000", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5067000000000005e-05 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.779812+0000", 2024-09-13T13:52:23.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7484000000000001e-05 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783760+0000", 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039480490000000003 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.783808+0000", 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7549999999999997e-05 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.462 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 81.561842373999994, 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:52:23.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 81.561074325999996, 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:52:23.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 76.204761046000002, 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:52:23.472 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3424 4.8 e118/34 4:1b24d731:::smithi15938319-13:head v 118'122, mlcod=115'119)", 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.428153821999999, 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041392160000000002, 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.477 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.982608+0000", 2024-09-13T13:52:23.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.34e-07 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983037+0000", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00042880899999999999 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983075+0000", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7994000000000002e-05 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983083+0000", 2024-09-13T13:52:23.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9030000000000004e-06 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983136+0000", 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3353999999999998e-05 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.983152+0000", 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5583999999999999e-05 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986717+0000", 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035656059999999998 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:35.986747+0000", 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9532000000000001e-05 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.481 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 45.997353781000001, 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:52:23.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 43.104668801999999, 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:52:23.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:52:23.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:52:23.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.453077151000002, 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:52:23.490 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4198 4.8 e144/34 4:165e03eb:::smithi15938319-32:head v 144'170, mlcod=144'168)", 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.299305545999999, 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041002770000000003, 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111456+0000", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111457+0000", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.111960+0000", 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00050279399999999998 2024-09-13T13:52:23.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112261+0000", 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00030132899999999997 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112267+0000", 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.139e-06 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112323+0000", 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5773999999999998e-05 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.112346+0000", 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3425999999999999e-05 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.115519+0000", 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031729390000000001 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:07.115556+0000", 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6854999999999999e-05 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:256 4.8 e150/34 4:165e03eb:::smithi15938319-32:2c v 150'186, mlcod=148'183)", 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.653766114, 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045821789999999996, 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.499 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756996+0000", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.98e-07 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757009+0000", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2948e-05 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757049+0000", 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0793000000000001e-05 2024-09-13T13:52:23.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757058+0000", 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8380000000000008e-06 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757115+0000", 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6483000000000001e-05 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757139+0000", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4167000000000001e-05 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761524+0000", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004384888 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761578+0000", 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3863999999999998e-05 2024-09-13T13:52:23.503 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:257 4.8 e150/34 4:1b24d731:::smithi15938319-13:2c v 150'188, mlcod=148'183)", 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.652725427, 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063443070000000004, 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.504 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758037+0000", 2024-09-13T13:52:23.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9999999999999996e-07 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758044+0000", 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.759e-06 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758116+0000", 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1913000000000001e-05 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758123+0000", 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.763e-06 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758187+0000", 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3283999999999999e-05 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758211+0000", 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4728000000000001e-05 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764325+0000", 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006114113 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764380+0000", 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4947000000000001e-05 2024-09-13T13:52:23.508 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:262 4.8 e154/34 4:1b24d731:::smithi15938319-13:26 v 154'194, mlcod=150'186)", 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 6.6447833909999998, 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058817069999999999, 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765979+0000", 2024-09-13T13:52:23.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1999999999999999e-07 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765984+0000", 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766015+0000", 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1711000000000002e-05 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766021+0000", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0009999999999998e-06 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766125+0000", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103187 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766138+0000", 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3067999999999999e-05 2024-09-13T13:52:23.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771829+0000", 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056913909999999996 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771860+0000", 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0787999999999997e-05 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.513 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:23.514 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:23.514 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:23.515 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:23.515 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:23.515 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:23.516 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:23.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:23.517 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:23.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:23.517 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:23.517 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:23.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:23.518 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:23.518 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:23.519 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:23.514+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:23.523 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:23.523 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:23.545 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:23.545 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:23.566 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:23.586 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:23.587 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:23.608 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:23.616 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:23.615+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:23.629 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:23.630 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:23.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:23.649+0000 7efd71c44640 0 -- 172.21.15.27:0/2231532669 >> v2:172.21.15.159:6805/4073091144 conn(0x7efd6c2202f0 msgr2=0x7efd6c2226e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:23.657 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:23.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:23.663 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:23.663 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:23.672 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:23.716 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:23.715+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:23.750 INFO:tasks.rados.rados.0.smithi159.stdout:1925: done (1 left) 2024-09-13T13:52:23.750 INFO:tasks.rados.rados.0.smithi159.stdout:1927: done (0 left) 2024-09-13T13:52:23.750 INFO:tasks.rados.rados.0.smithi159.stdout:1928: write oid 16 current snap is 57 2024-09-13T13:52:23.751 INFO:tasks.rados.rados.0.smithi159.stdout:1928: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-13T13:52:23.758 INFO:tasks.rados.rados.0.smithi159.stdout:1928: writing smithi15938319-16 from 702625 to 1422297 tid 1 2024-09-13T13:52:23.763 INFO:tasks.rados.rados.0.smithi159.stdout:1928: writing smithi15938319-16 from 2127571 to 2302625 tid 2 2024-09-13T13:52:23.763 INFO:tasks.rados.rados.0.smithi159.stdout:1929: delete oid 30 current snap is 57 2024-09-13T13:52:23.771 INFO:tasks.rados.rados.0.smithi159.stdout:1929: done (1 left) 2024-09-13T13:52:23.771 INFO:tasks.rados.rados.0.smithi159.stdout:1930: set_chunk oid 46 target oid 2024-09-13T13:52:23.772 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:23.771+0000 7fe859f53640 -1 osd.7 159 handle_osd_map injecting map crc failure 2024-09-13T13:52:23.775 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:23.776 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:23.778 INFO:tasks.rados.rados.0.smithi159.stdout:1930: set_chunk oid 46 offset: 641024 length: 67584 target oid 864b175f1ecf87d07bab9a68f9e7f1338ec93e2030416ae1312f0baaacf0c1ed offset: 0 snap -1 2024-09-13T13:52:23.778 INFO:tasks.rados.rados.0.smithi159.stdout:1931: set_chunk oid 4 target oid 2024-09-13T13:52:23.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:23.787 INFO:tasks.rados.rados.0.smithi159.stdout:1931: set_chunk oid 4 offset: 1956352 length: 555520 target oid e0ba1a5d0dec6307b0abd8a040e5f5481e729ffd17f7c4666fc29164083c8fa2 offset: 0 snap -1 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1932: tier_evict oid 32 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1932: tier_evict oid 32 snap -1 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: finishing write tid 1 to smithi15938319-16 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1933: read oid 6 snap -1 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1933: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1934: read oid 10 snap -1 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1934: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1935: write (excl) oid 27 current snap is 57 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: finishing write tid 2 to smithi15938319-16 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: finishing write tid 3 to smithi15938319-16 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: finishing write tid 4 to smithi15938319-16 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: oid 16 updating version 0 to 443 2024-09-13T13:52:23.788 INFO:tasks.rados.rados.0.smithi159.stdout:1928: oid 16 version 443 is already newer than 442 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:1928: oid 16 version 443 is already newer than 441 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 443 (ObjNum 375 snap 57 seq_num 375) dirty exists 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:1928: left oid 16 (ObjNum 375 snap 57 seq_num 375) 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:1930: finishing set_chunk to oid 46 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 46 v 154 (ObjNum 345 snap 47 seq_num 345) dirty exists 2024-09-13T13:52:23.789 INFO:tasks.rados.rados.0.smithi159.stdout:1935: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-13T13:52:23.792 INFO:tasks.rados.rados.0.smithi159.stdout:1935: writing smithi15938319-27 from 736239 to 1206409 tid 1 2024-09-13T13:52:23.798 INFO:tasks.rados.rados.0.smithi159.stdout:1935: writing smithi15938319-27 from 1728182 to 2299964 tid 2 2024-09-13T13:52:23.798 INFO:tasks.rados.rados.0.smithi159.stdout:1928: done (6 left) 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:1930: done (5 left) 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:1936: snap_create 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:1932: got (22) Invalid argument 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:1931: finishing set_chunk to oid 4 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 4 v 293 (ObjNum 320 snap 46 seq_num 320) dirty exists 2024-09-13T13:52:23.799 INFO:tasks.rados.rados.0.smithi159.stdout:1934: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:23.816 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:23.815+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:23.845 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:23.895 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:23.895 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:23.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:23.936 INFO:tasks.rados.rados.0.smithi159.stdout:1933: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:23.937 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:23.937 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:23.935+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:24.011 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.011 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:24.020 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:24.018+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: finishing write tid 1 to smithi15938319-27 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: finishing write tid 2 to smithi15938319-27 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: finishing write tid 3 to smithi15938319-27 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: finishing write tid 4 to smithi15938319-27 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: oid 27 updating version 0 to 352 2024-09-13T13:52:24.073 INFO:tasks.rados.rados.0.smithi159.stdout:1935: oid 27 version 352 is already newer than 351 2024-09-13T13:52:24.074 INFO:tasks.rados.rados.0.smithi159.stdout:1935: oid 27 updating version 352 to 353 2024-09-13T13:52:24.074 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 353 (ObjNum 377 snap 57 seq_num 377) dirty exists 2024-09-13T13:52:24.074 INFO:tasks.rados.rados.0.smithi159.stdout:1935: left oid 27 (ObjNum 377 snap 57 seq_num 377) 2024-09-13T13:52:24.114 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:24.114+0000 7f4ffbd7a640 -1 osd.1 159 handle_osd_map injecting map crc failure 2024-09-13T13:52:24.120 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:24.120+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:24.127 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.127 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:24.222 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:24.222+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:24.244 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.244 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:24.322 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:24.322+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:24.359 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.360 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:24.423 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:24.422+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:24.476 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.476 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:24.529 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:24.528+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:24.591 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.592 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:24.642 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:24.640+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:24.725 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:24.724+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:24.748 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.748 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.753 INFO:tasks.rados.rados.0.smithi159.stdout:1931: done (5 left) 2024-09-13T13:52:24.753 INFO:tasks.rados.rados.0.smithi159.stdout:1932: done (4 left) 2024-09-13T13:52:24.753 INFO:tasks.rados.rados.0.smithi159.stdout:1933: done (3 left) 2024-09-13T13:52:24.753 INFO:tasks.rados.rados.0.smithi159.stdout:1934: done (2 left) 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1935: done (1 left) 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1936: done (0 left) 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1937: read oid 2 snap 49 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1937: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1938: read oid 9 snap 34 2024-09-13T13:52:24.754 INFO:tasks.rados.rados.0.smithi159.stdout:1938: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:24.755 INFO:tasks.rados.rados.0.smithi159.stdout:1939: write (excl) oid 47 current snap is 58 2024-09-13T13:52:24.755 INFO:tasks.rados.rados.0.smithi159.stdout:1939: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-13T13:52:24.756 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:24.751+0000 7fe0fbe07640 -1 osd.2 160 handle_osd_map injecting map crc failure 2024-09-13T13:52:24.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:24.759 INFO:tasks.rados.rados.0.smithi159.stdout:1939: writing smithi15938319-47 from 753046 to 1298465 tid 1 2024-09-13T13:52:24.764 INFO:tasks.rados.rados.0.smithi159.stdout:1939: writing smithi15938319-47 from 1786664 to 2218457 tid 2 2024-09-13T13:52:24.764 INFO:tasks.rados.rados.0.smithi159.stdout:1940: read oid 49 snap -1 2024-09-13T13:52:24.764 INFO:tasks.rados.rados.0.smithi159.stdout:1940: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:52:24.764 INFO:tasks.rados.rados.0.smithi159.stdout:1941: snap_create 2024-09-13T13:52:24.772 INFO:tasks.rados.rados.0.smithi159.stdout:1939: finishing write tid 1 to smithi15938319-47 2024-09-13T13:52:24.772 INFO:tasks.rados.rados.0.smithi159.stdout:1939: finishing write tid 2 to smithi15938319-47 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: finishing write tid 3 to smithi15938319-47 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: finishing write tid 4 to smithi15938319-47 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: oid 47 updating version 0 to 293 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: oid 47 updating version 293 to 294 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: oid 47 version 294 is already newer than 292 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 47 v 294 (ObjNum 378 snap 58 seq_num 378) dirty exists 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1939: left oid 47 (ObjNum 378 snap 58 seq_num 378) 2024-09-13T13:52:24.773 INFO:tasks.rados.rados.0.smithi159.stdout:1938: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:24.774 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 200344 2024-09-13T13:52:24.822 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:24.826 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:24.825+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:24.872 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:24.885 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:24.885+0000 7f4ffdd7e640 -1 osd.1 160 handle_osd_map injecting map crc failure 2024-09-13T13:52:24.905 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:24.905 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:24.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:24.922 INFO:tasks.rados.rados.0.smithi159.stdout:1937: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:24.957 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:24.956+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:24.973 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2899174 2024-09-13T13:52:24.987 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:25.021 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.021 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.027 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:25.027+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:25.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:25.128 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:25.128+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:25.141 INFO:tasks.rados.rados.0.smithi159.stdout:1940: expect (ObjNum 173 snap 16 seq_num 173) 2024-09-13T13:52:25.145 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.145 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:25.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:52:25.183 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:25.230 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:25.229+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:25.256 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.256 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:25.331 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:25.330+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:25.367 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.368 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:25.432 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:25.431+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:25.480 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.480 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:25.572 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:25.572+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:25.631 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.632 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.634 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:25.633+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:25.640 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:25.662 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:25.734 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:25.733+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:25.749 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1937: done (4 left) 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1938: done (3 left) 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1939: done (2 left) 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1940: done (1 left) 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1941: done (0 left) 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1942: read oid 26 snap -1 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1942: expect deleted 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1943: write oid 33 current snap is 59 2024-09-13T13:52:25.755 INFO:tasks.rados.rados.0.smithi159.stdout:1943: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-13T13:52:25.762 INFO:tasks.rados.rados.0.smithi159.stdout:1943: writing smithi15938319-33 from 769853 to 1390521 tid 1 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1943: writing smithi15938319-33 from 1845146 to 2369853 tid 2 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1944: read oid 6 snap -1 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1944: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1945: read oid 40 snap -1 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1945: expect (ObjNum 371 snap 54 seq_num 371) 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1946: read oid 12 snap 58 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1946: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1947: write oid 42 current snap is 59 2024-09-13T13:52:25.768 INFO:tasks.rados.rados.0.smithi159.stdout:1947: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-13T13:52:25.769 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:25.768+0000 7f9b0e557640 -1 osd.12 161 handle_osd_map injecting map crc failure 2024-09-13T13:52:25.775 INFO:tasks.rados.rados.0.smithi159.stdout:1947: writing smithi15938319-42 from 786660 to 1482577 tid 1 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1947: writing smithi15938319-42 from 1903628 to 2339090 tid 2 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1942: done (5 left) 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1948: tier_evict oid 32 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1948: tier_evict oid 32 snap -1 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: finishing write tid 1 to smithi15938319-33 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: finishing write tid 2 to smithi15938319-33 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: finishing write tid 3 to smithi15938319-33 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: finishing write tid 4 to smithi15938319-33 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: oid 33 updating version 0 to 310 2024-09-13T13:52:25.781 INFO:tasks.rados.rados.0.smithi159.stdout:1943: oid 33 updating version 310 to 312 2024-09-13T13:52:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1943: oid 33 version 312 is already newer than 311 2024-09-13T13:52:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 33 v 312 (ObjNum 379 snap 59 seq_num 379) dirty exists 2024-09-13T13:52:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1943: left oid 33 (ObjNum 379 snap 59 seq_num 379) 2024-09-13T13:52:25.782 INFO:tasks.rados.rados.0.smithi159.stdout:1945: expect (ObjNum 371 snap 54 seq_num 371) 2024-09-13T13:52:25.785 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.785 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.789 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 641798 2024-09-13T13:52:25.794 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:25.799 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:25.800 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:25.821 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:25.821 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:25.834 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:25.834+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:25.843 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:25.844 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:25.864 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:25.885 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:25.885 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:25.897 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:25.897 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:25.906 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:25.907 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:25.907 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:25.908 INFO:tasks.rados.rados.0.smithi159.stdout:1944: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:25.929 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:25.929 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1510940 2024-09-13T13:52:25.929 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:25.936 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:25.935+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:25.947 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 277356 2024-09-13T13:52:25.948 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:25.949 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:25.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:25.991 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:25.992 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:26.009 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:26.010 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:26.010 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:26.013 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:26.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:26.033 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:26.036 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:26.036+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1948: got (22) Invalid argument 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1947: finishing write tid 1 to smithi15938319-42 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1947: finishing write tid 2 to smithi15938319-42 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1943: done (5 left) 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1944: done (4 left) 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1945: done (3 left) 2024-09-13T13:52:26.041 INFO:tasks.rados.rados.0.smithi159.stdout:1948: done (2 left) 2024-09-13T13:52:26.042 INFO:tasks.rados.rados.0.smithi159.stdout:1949: write (excl) oid 41 current snap is 59 2024-09-13T13:52:26.042 INFO:tasks.rados.rados.0.smithi159.stdout:1949: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-13T13:52:26.045 INFO:tasks.rados.rados.0.smithi159.stdout:1949: writing smithi15938319-41 from 403467 to 890986 tid 1 2024-09-13T13:52:26.052 INFO:tasks.rados.rados.0.smithi159.stdout:1949: writing smithi15938319-41 from 1562110 to 2141230 tid 2 2024-09-13T13:52:26.052 INFO:tasks.rados.rados.0.smithi159.stdout:1950: set_chunk oid 45 target oid 2024-09-13T13:52:26.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:26.055 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1950: set_chunk oid 45 offset: 1047552 length: 210432 target oid e899a0ebcc87929f129918257af93528cea7a693a18fc95ecce72bc058d12671 offset: 0 snap 56 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1947: finishing write tid 3 to smithi15938319-42 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1951: read oid 3 snap -1 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1951: expect (ObjNum 350 snap 50 seq_num 350) 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1952: read oid 10 snap -1 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1952: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1953: write oid 30 current snap is 59 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1947: finishing write tid 4 to smithi15938319-42 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1947: oid 42 updating version 0 to 233 2024-09-13T13:52:26.057 INFO:tasks.rados.rados.0.smithi159.stdout:1947: oid 42 version 233 is already newer than 232 2024-09-13T13:52:26.058 INFO:tasks.rados.rados.0.smithi159.stdout:1947: oid 42 version 233 is already newer than 231 2024-09-13T13:52:26.058 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 233 (ObjNum 380 snap 59 seq_num 380) dirty exists 2024-09-13T13:52:26.058 INFO:tasks.rados.rados.0.smithi159.stdout:1947: left oid 42 (ObjNum 380 snap 59 seq_num 380) 2024-09-13T13:52:26.058 INFO:tasks.rados.rados.0.smithi159.stdout:1946: expect (ObjNum 322 snap 46 seq_num 322) 2024-09-13T13:52:26.058 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:26.077 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:26.077 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:26.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:26.098+0000 7f6ec9d2b640 0 -- 172.21.15.27:0/3452124969 >> v2:172.21.15.159:6805/4073091144 conn(0x7f6ec4220350 msgr2=0x7f6ec4222740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:26.106 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:26.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 154.887157704, 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.122 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:52:26.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:52:26.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 153.879123558, 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.127 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:52:26.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:52:26.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 104.483425399, 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:52:26.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.135 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.936521697000003, 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:52:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:52:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.140 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.931680873000005, 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:52:26.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:52:26.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.930967898000006, 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.145 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:52:26.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.149 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:26.150 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2349268 2024-09-13T13:52:26.151 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:26.150+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:26.151 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:26.152 INFO:tasks.rados.rados.0.smithi159.stdout:1953: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-13T13:52:26.157 INFO:tasks.rados.rados.0.smithi159.stdout:1953: writing smithi15938319-30 from 420274 to 983042 tid 1 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1953: writing smithi15938319-30 from 1620592 to 2059723 tid 2 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1946: done (6 left) 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1947: done (5 left) 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1954: read oid 6 snap -1 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1954: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1949: finishing write tid 1 to smithi15938319-41 2024-09-13T13:52:26.164 INFO:tasks.rados.rados.0.smithi159.stdout:1955: set_chunk oid 33 target oid 2024-09-13T13:52:26.168 INFO:tasks.rados.rados.0.smithi159.stdout:1955: set_chunk oid 33 offset: 2543616 length: 227328 target oid 52376603202ed123fc7693e75eb5d8b33e9c66d1951074004424d233a7f997cf offset: 0 snap 57 2024-09-13T13:52:26.168 INFO:tasks.rados.rados.0.smithi159.stdout:1956: snap_remove snap 34 2024-09-13T13:52:26.251 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:26.251+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:26.254 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:26.254 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:26.255 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:26.255 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:26.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:26.352 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:26.351+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:26.406 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:26.407 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:26.415 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:26.453 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:26.452+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.75069186900001, 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:52:26.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:52:26.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.74971467099999, 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:26.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:52:26.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:52:26.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.747960006, 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:52:26.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:52:26.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.747567793, 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.747530466, 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.545 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:52:26.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:52:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.74273211299999, 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:52:26.551 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:52:26.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:52:26.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.72709688800001, 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:52:26.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:52:26.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.564 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.726990538, 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:52:26.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:52:26.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.570 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.719056768, 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.571 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:26.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:52:26.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.70066821, 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.577 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:52:26.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:52:26.583 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.70054142000001, 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:52:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.61934299699999, 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:26.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:52:26.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.61924154600001, 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:52:26.596 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:52:26.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 122.05788878, 2024-09-13T13:52:26.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:26.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 122.05203723699999, 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:52:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:52:26.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:52:26.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:26.612 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 112.790159786, 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.619 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 111.78413851099999, 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:52:26.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:52:26.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.7.23:20 3.1 3:8dbd923e:::ae8b86d07d375a083683440b6bda0e0f67b5195d585764d7ac407424ac114e59:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.500703338000001, 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89571672999999996, 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.625 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016793+0000", 2024-09-13T13:52:26.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6600000000000005e-07 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016800+0000", 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1199999999999996e-06 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910876+0000", 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89407579400000003 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910884+0000", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910982+0000", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7763999999999994e-05 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911101+0000", 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119942 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911338+0000", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023690000000000001 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912020+0000", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068117299999999998 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912443+0000", 2024-09-13T13:52:26.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042320999999999998 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912478+0000", 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4768999999999999e-05 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912509+0000", 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1226999999999998e-05 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(3.1 126/125 [PushReplyOp(3:ac2af9f5:::e6b4f9f8265bfe5a786cb2bd77c53a79b3e484041c7c0186a813c80a2460efe6:head)])", 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.471284527000002, 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208631544, 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:26.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9399999999999995e-07 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046216+0000", 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3959999999999999e-06 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254500+0000", 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208283991 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254505+0000", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.789e-06 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254594+0000", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8101000000000002e-05 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254842+0000", 2024-09-13T13:52:26.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024877299999999998 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.6.22:64 3.1 3:92340f89:::0d01361f24984ff4b0b1eb9d9ead6642f5b878662a350c2f914a3a82940a2cbd:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e148)", 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 15.178412561, 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.044891857, 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.635 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:26.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339084+0000", 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.251e-06 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339090+0000", 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382252+0000", 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043162303999999999 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382274+0000", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1345999999999999e-05 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382467+0000", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019345899999999999 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382593+0000", 2024-09-13T13:52:26.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012580600000000001 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 14", 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382745+0000", 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015200200000000001 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383376+0000", 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063144299999999998 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383909+0000", 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053270499999999996 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383947+0000", 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7901000000000002e-05 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383975+0000", 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.777e-05 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:26.641 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:26.641 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:26.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:26.643 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:26.642+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:26.742 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:26.742+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:26.758 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:26.758 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:26.758 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:26.758 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:26.767 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:26.844 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:26.843+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:26.871 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:26.871 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:26.872 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:26.872 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:26.872 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:26.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:26.945 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:26.945+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.984 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 134.08417103100001, 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:52:26.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:52:26.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:26.990 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 134.06854341600001, 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:52:26.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:52:26.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:26.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:52:26.996 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 134.05983174100001, 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:26.997 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:26.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:26.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:52:27.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 118.235545591, 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.003 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:52:27.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 117.020875312, 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:52:27.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 114.019073686, 2024-09-13T13:52:27.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:27.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:52:27.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.74015011900001, 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.018 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:52:27.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.023 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 105.74012676, 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:27.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:52:27.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 86.15606056, 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.029 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:52:27.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.221190316000005, 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:52:27.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.125246000999994, 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.040 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:52:27.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 75.125208139999998, 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.046 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:52:27.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:27.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2905 4.1 4:8b8235b2:::smithi15938319-16:head [write 1577796~455884] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.144513269000001, 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010065592, 2024-09-13T13:52:27.051 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2905 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837823+0000", 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.837825+0000", 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.285e-06 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838182+0000", 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035747300000000002 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838197+0000", 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4755e-05 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838203+0000", 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1550000000000004e-06 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838251+0000", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8010000000000003e-05 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838340+0000", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8688000000000002e-05 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838506+0000", 2024-09-13T13:52:27.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016683400000000001 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840251+0000", 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017447039999999999 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847770+0000", 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075189499999999999 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847829+0000", 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302000000000002e-05 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847889+0000", 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9435999999999998e-05 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2906 4.1 4:8b8235b2:::smithi15938319-16:head [setxattr _header (57) in=7b,truncate 2033680] snapc 1b=[1b,1a,19,18,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.144120545000007, 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098570229999999995, 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.058 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2906 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838216+0000", 2024-09-13T13:52:27.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838217+0000", 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-07 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838219+0000", 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7930000000000002e-06 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838223+0000", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.924e-06 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838227+0000", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.258e-06 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839399+0000", 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001172394 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839482+0000", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3362999999999995e-05 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.839586+0000", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010336399999999999 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840535+0000", 2024-09-13T13:52:27.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094923800000000003 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.847957+0000", 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074225530000000001 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848014+0000", 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6328e-05 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848073+0000", 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9233e-05 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:27.064 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 74.144104944999995, 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:52:27.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.069 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3899 4.1 4:8bb695d2:::smithi15938319-48:head [write 561433~774904,stat] snapc 2f=[2f,2d,2c,2a,28,27,25,24,23,22,1e,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 29.687557506000001, 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011332106, 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.070 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3899 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294781+0000", 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295899+0000", 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011175060000000001 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295916+0000", 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7275e-05 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295925+0000", 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2550000000000003e-06 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295992+0000", 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6973999999999999e-05 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295999+0000", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299822+0000", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038227019999999999 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299902+0000", 2024-09-13T13:52:27.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0341999999999998e-05 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.300046+0000", 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014437299999999999 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.304078+0000", 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040313459999999999 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.305972+0000", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018946200000000001 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306041+0000", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8399999999999996e-05 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306111+0000", 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0492000000000002e-05 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.076 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4169 4.1 4:81ab2613:::smithi15938319-29:head [read 0~3462209,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.895388830999998, 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012063419000000001, 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.077 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4169 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086949+0000", 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086957+0000", 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6350000000000006e-06 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086972+0000", 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5503e-05 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086982+0000", 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0009e-05 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087052+0000", 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9832999999999994e-05 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087058+0000", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091635+0000", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045774259999999999 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091765+0000", 2024-09-13T13:52:27.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012954199999999999 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099011+0000", 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072461840000000001 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4170 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.895331023000001, 2024-09-13T13:52:27.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014864408000000001, 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4170 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:27.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087006+0000", 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0100000000000005e-07 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087008+0000", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4200000000000001e-06 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087015+0000", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087018+0000", 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7359999999999999e-06 2024-09-13T13:52:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087126+0000", 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107847 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087134+0000", 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2849999999999995e-06 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099058+0000", 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011923375 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099212+0000", 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015423100000000001 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101870+0000", 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002657899 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.087 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4171 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 20.895295398999998, 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016120193000000001, 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.088 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4171 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3700000000000001e-07 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087044+0000", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087048+0000", 2024-09-13T13:52:27.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3170000000000003e-06 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087053+0000", 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.609e-06 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087165+0000", 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111686 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087169+0000", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6600000000000001e-06 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101916+0000", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014746795 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.102043+0000", 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012732600000000001 2024-09-13T13:52:27.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.103161+0000", 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011183759999999999 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4390 4.3 4:c1ae59bf:::smithi15938319-45:head [write 416906~407733,stat] snapc 35=[35,34,33,32,31,2d,25,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:15.785817+0000", 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 11.19651973, 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098808339999999998, 2024-09-13T13:52:27.093 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4390 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.785817+0000", 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.785817+0000", 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.785819+0000", 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9250000000000002e-06 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.786903+0000", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010845710000000001 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.786920+0000", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7093000000000001e-05 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.786930+0000", 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.787006+0000", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6044000000000006e-05 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.787013+0000", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9519999999999997e-06 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.789429+0000", 2024-09-13T13:52:27.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024166790000000001 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.789523+0000", 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3746000000000007e-05 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 11", 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.789724+0000", 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0002008 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.791686+0000", 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001961707 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.795626+0000", 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039403179999999999 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.795667+0000", 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1579000000000003e-05 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:15.795698+0000", 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0190999999999999e-05 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.100 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:27.102 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:27.101+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:27.170 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:27.169+0000 7f002fbdf640 -1 osd.15 162 handle_osd_map injecting map crc failure 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:1956: done (7 left) 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:1957: read oid 39 snap 56 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:1957: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:1958: rollback oid 32 current snap is 59 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 32 to 33 2024-09-13T13:52:27.175 INFO:tasks.rados.rados.0.smithi159.stdout:1949: finishing write tid 2 to smithi15938319-41 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: finishing write tid 3 to smithi15938319-41 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: finishing write tid 4 to smithi15938319-41 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: oid 41 updating version 0 to 339 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: oid 41 updating version 339 to 340 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: oid 41 version 340 is already newer than 338 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 340 (ObjNum 381 snap 59 seq_num 381) dirty exists 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1949: left oid 41 (ObjNum 381 snap 59 seq_num 381) 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:1952: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:27.176 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 597166 2024-09-13T13:52:27.202 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:27.202+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:27.212 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.212 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:27.212 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:27.212 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:27.297 INFO:tasks.rados.rados.0.smithi159.stdout:1950: finishing set_chunk to oid 45 2024-09-13T13:52:27.297 INFO:tasks.rados.rados.0.smithi159.stdout:1951: expect (ObjNum 350 snap 50 seq_num 350) 2024-09-13T13:52:27.303 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:27.302+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:27.304 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1078553 2024-09-13T13:52:27.324 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:27.324 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.333 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:27.378 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:27.403 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:27.403+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:27.466 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:27.478 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.478 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:27.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:27.478 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 158.934878588, 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:52:27.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:52:27.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.483 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 156.88055269899999, 2024-09-13T13:52:27.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:27.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:52:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:52:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 156.86717430900001, 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:52:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 129.05384137199999, 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.24842936500001, 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:52:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:52:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:52:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.248392193, 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:52:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.24837146500001, 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:52:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.24827516400001, 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:52:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:52:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.24825251999999, 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:52:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.24822648999999, 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:52:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.108725623, 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:52:27.537 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:27.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:52:27.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:52:27.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.10861867600001, 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.544 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:52:27.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:52:27.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.108589417, 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.550 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:27.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:52:27.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.102225014, 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.556 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:52:27.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:52:27.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:27.566 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 119.09726482000001, 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:52:27.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:52:27.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:52:27.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.15527510299999, 2024-09-13T13:52:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:52:27.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.581 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.155235965, 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:52:27.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:52:27.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 85.613805967999994, 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:52:27.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 85.613743021999994, 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.592 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:52:27.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:52:27.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 85.61370934, 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.597 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:52:27.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.602 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.603 INFO:tasks.rados.rados.0.smithi159.stdout:1953: finishing write tid 1 to smithi15938319-30 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: finishing write tid 2 to smithi15938319-30 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: finishing write tid 3 to smithi15938319-30 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: finishing write tid 4 to smithi15938319-30 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: oid 30 updating version 0 to 356 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: oid 30 version 356 is already newer than 355 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: oid 30 version 356 is already newer than 354 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 30 v 356 (ObjNum 382 snap 59 seq_num 382) dirty exists 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1953: left oid 30 (ObjNum 382 snap 59 seq_num 382) 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:1954: expect (ObjNum 253 snap 28 seq_num 253) 2024-09-13T13:52:27.604 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1089610 2024-09-13T13:52:27.605 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:27.602+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:27.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:27.690 INFO:tasks.rados.rados.0.smithi159.stdout:1955: finishing set_chunk to oid 33 2024-09-13T13:52:27.690 INFO:tasks.rados.rados.0.smithi159.stdout:1957: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:27.704 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:27.703+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:27.708 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.708 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:27.708 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:27.708 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:27.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:52:27.775 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2449133 2024-09-13T13:52:27.805 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:27.804+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:27.821 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.821 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:27.821 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:27.821 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:27.821 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:27.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:1958: finishing rollback tid 1 to smithi15938319-32 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:1958: finishing rollback tid 2 to smithi15938319-32 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 206 (ObjNum 220 snap 24 seq_num 220) dirty exists 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:1949: done (8 left) 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:1950: done (7 left) 2024-09-13T13:52:27.867 INFO:tasks.rados.rados.0.smithi159.stdout:1951: done (6 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1952: done (5 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1953: done (4 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1954: done (3 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1955: done (2 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1957: done (1 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1958: done (0 left) 2024-09-13T13:52:27.868 INFO:tasks.rados.rados.0.smithi159.stdout:1959: snap_create 2024-09-13T13:52:27.905 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:27.905+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:27.974 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.235723339, 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.975 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:52:27.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:52:27.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.230482717, 2024-09-13T13:52:27.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:27.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:52:27.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:52:27.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.23043140300001, 2024-09-13T13:52:27.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:52:27.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.993 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.230359206, 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:27.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:27.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:52:27.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:52:27.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:27.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.23032020100001, 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.000 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:52:28.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:52:28.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.23025578900001, 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.006 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:52:28.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.204905693, 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.012 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:52:28.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:52:28.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:52:28.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.17434486900001, 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:52:28.018 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:52:28.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.024 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.15301828200001, 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:52:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:52:28.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.07480200500001, 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.029 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:52:28.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:52:28.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.034 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.730830345, 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.035 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:28.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:52:28.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:52:28.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.730772293, 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:52:28.041 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:52:28.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:52:28.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.730746334, 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.049 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:52:28.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:52:28.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.730631918, 2024-09-13T13:52:28.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:28.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:52:28.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:52:28.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.062 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.73060102700001, 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.063 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:28.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:52:28.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:52:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:52:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.730564444, 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.071 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:52:28.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:52:28.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:52:28.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.65076957399999, 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.078 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:52:28.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:28.082 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.276109156, 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:52:28.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:28.086 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 87.152917633000001, 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:52:28.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.092 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 87.144897334999996, 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:52:28.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:52:28.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.096 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.097 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.097 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.097 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:28.098 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:28.099 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:28.098+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:28.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:28.172 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:52:28.175 INFO:tasks.rados.rados.0.smithi159.stdout:1959: done (0 left) 2024-09-13T13:52:28.175 INFO:tasks.rados.rados.0.smithi159.stdout:1960: write oid 34 current snap is 60 2024-09-13T13:52:28.175 INFO:tasks.rados.rados.0.smithi159.stdout:1960: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-13T13:52:28.181 INFO:tasks.rados.rados.0.smithi159.stdout:1960: writing smithi15938319-34 from 437081 to 1075098 tid 1 2024-09-13T13:52:28.188 INFO:tasks.rados.rados.0.smithi159.stdout:1960: writing smithi15938319-34 from 1679074 to 2261863 tid 2 2024-09-13T13:52:28.188 INFO:tasks.rados.rados.0.smithi159.stdout:1961: write (excl) oid 3 current snap is 60 2024-09-13T13:52:28.188 INFO:tasks.rados.rados.0.smithi159.stdout:1961: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-13T13:52:28.193 INFO:tasks.rados.rados.0.smithi159.stdout:1961: writing smithi15938319-3 from 453888 to 1167154 tid 1 2024-09-13T13:52:28.199 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:28.198+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1961: writing smithi15938319-3 from 1737556 to 2180356 tid 2 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1962: read oid 45 snap 49 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1962: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1963: write oid 15 current snap is 60 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: finishing write tid 1 to smithi15938319-34 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: finishing write tid 2 to smithi15938319-34 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: finishing write tid 3 to smithi15938319-34 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: finishing write tid 4 to smithi15938319-34 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: oid 34 updating version 0 to 304 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: oid 34 updating version 304 to 306 2024-09-13T13:52:28.200 INFO:tasks.rados.rados.0.smithi159.stdout:1960: oid 34 version 306 is already newer than 305 2024-09-13T13:52:28.201 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 34 v 306 (ObjNum 383 snap 60 seq_num 383) dirty exists 2024-09-13T13:52:28.201 INFO:tasks.rados.rados.0.smithi159.stdout:1960: left oid 34 (ObjNum 383 snap 60 seq_num 383) 2024-09-13T13:52:28.201 INFO:tasks.rados.rados.0.smithi159.stdout:1963: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-13T13:52:28.206 INFO:tasks.rados.rados.0.smithi159.stdout:1963: writing smithi15938319-15 from 470695 to 1259210 tid 1 2024-09-13T13:52:28.208 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:28.208 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:28.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.208 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4611 4.b 4:def5de2b:::smithi15938319-3:head [write 453888~713266 in=713266b,stat out=16b] snapc 3d=[3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,2d,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:28.208 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.194318+0000", 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.013785041, 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013805423000000001, 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4611 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.194318+0000", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.194318+0000", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.194319+0000", 2024-09-13T13:52:28.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.513e-06 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.195025+0000", 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070558100000000005 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.195040+0000", 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4866e-05 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.195048+0000", 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9990000000000008e-06 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.195097+0000", 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9008000000000002e-05 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.195101+0000", 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8750000000000002e-06 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197072+0000", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001970861 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197135+0000", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3514000000000002e-05 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197250+0000", 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114585 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199668+0000", 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002418828 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4612 4.b 4:def5de2b:::smithi15938319-3:head [write 1737556~442800 in=442800b] snapc 3d=[3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,2d,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.199784+0000", 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0083190460000000001, 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083887470000000002, 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:52:28.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4612 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199784+0000", 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199784+0000", 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199785+0000", 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1289999999999999e-06 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200221+0000", 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00043574199999999998 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200237+0000", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6252e-05 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200245+0000", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2879999999999993e-06 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200285+0000", 2024-09-13T13:52:28.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0256000000000002e-05 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200344+0000", 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8955000000000001e-05 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200446+0000", 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101882 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201819+0000", 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001372346 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4613 4.b 4:def5de2b:::smithi15938319-3:head [setxattr _header (56) in=63b,truncate 2453888] snapc 3d=[3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,2d,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.200261+0000", 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0078413730000000004, 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079506839999999995, 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.219 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4613 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200261+0000", 2024-09-13T13:52:28.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200261+0000", 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200262+0000", 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9699999999999995e-07 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200266+0000", 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6009999999999999e-06 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200271+0000", 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4070000000000001e-06 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200277+0000", 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6690000000000003e-06 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201231+0000", 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095435599999999995 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201307+0000", 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6106999999999995e-05 2024-09-13T13:52:28.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201408+0000", 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010077399999999999 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.204223+0000", 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028145119999999999 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.224 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4614 4.b 4:def5de2b:::smithi15938319-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.200286+0000", 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0078164789999999994, 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079620190000000007, 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for rw locks", 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4614 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200286+0000", 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200286+0000", 2024-09-13T13:52:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200287+0000", 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200289+0000", 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.08e-06 2024-09-13T13:52:28.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200295+0000", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2559999999999999e-06 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.200300+0000", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6179999999999997e-06 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201849+0000", 2024-09-13T13:52:28.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015491159999999999 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.201900+0000", 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1054000000000003e-05 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 4 2024-09-13T13:52:28.229 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1963: writing smithi15938319-15 from 1679685 to 2266143 tid 2 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1960: done (3 left) 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1964: write oid 41 current snap is 60 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1964: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1964: writing smithi15938319-41 from 487502 to 951266 tid 1 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1964: writing smithi15938319-41 from 1738167 to 2184636 tid 2 2024-09-13T13:52:28.231 INFO:tasks.rados.rados.0.smithi159.stdout:1965: snap_remove snap 51 2024-09-13T13:52:28.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:28.249 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:28.249 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:28.272 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:28.293 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:28.299 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:28.299+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:28.314 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:28.315 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:28.335 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:28.335 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:28.335 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:28.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:28.401 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:28.400+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:28.489 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 158.25939351599999, 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:52:28.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:52:28.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:52:28.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.75465533400001, 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:52:28.497 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:28.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:52:28.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.503 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.75460602199999, 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:52:28.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.510 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.754590318, 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:28.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:52:28.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.516 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.754562642, 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.517 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.71934504399999, 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.523 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:52:28.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.712276062, 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:52:28.529 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:52:28.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.6777376, 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:52:28.535 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:52:28.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:52:28.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:52:28.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.541 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.670107758, 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:52:28.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:52:28.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:52:28.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.547 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.66673138600001, 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:52:28.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:52:28.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:52:28.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:52:28.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 135.56387210599999, 2024-09-13T13:52:28.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 131.35284856600001, 2024-09-13T13:52:28.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:52:28.563 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 131.34133328300001, 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:52:28.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.240024815, 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:52:28.568 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:52:28.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:28.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:52:28.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.239960467, 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:52:28.575 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:52:28.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:52:28.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:52:28.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.239932405, 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.581 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:52:28.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:52:28.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:28.586 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.238250515, 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:52:28.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:52:28.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.79872230700001, 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:28.594 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:52:28.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:52:28.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.597 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 87.670086552000001, 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:52:28.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.604 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:52:28.604 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:28.604 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 57.524829951999997, 2024-09-13T13:52:28.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:28.605 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:52:28.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:52:28.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:28.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:52:28.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:28.615 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:28.615 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:28.616 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:28.616 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:28.617 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:28.617 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:28.618 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:28.618 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:28.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:28.619 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:28.619 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:28.619 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:28.616+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:28.623 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:28.624 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:28.645 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:28.645 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:28.666 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:28.666 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:28.686 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:28.708 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:28.708 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:28.717 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:28.717+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:28.723 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:28.723 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:28.729 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:28.729 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:28.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:28.729+0000 7fa582ffd640 0 -- 172.21.15.27:0/3613441600 >> v2:172.21.15.159:6805/4073091144 conn(0x7fa584220720 msgr2=0x7fa584222b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:28.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:28.818 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:28.817+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:28.835 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:28.835 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:28.843 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:28.919 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:28.919+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:28.947 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:28.948 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:28.953 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:28.953 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:28.957 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:28.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:29.020 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:29.019+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:29.062 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.062 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.070 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:29.121 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:29.120+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:29.173 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.173 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.180 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:29.179+0000 7f8f5ac65640 -1 osd.6 164 handle_osd_map injecting map crc failure 2024-09-13T13:52:29.182 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1965: done (4 left) 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1966: tier_evict oid 26 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1966: tier_evict oid 26 snap -1 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1961: finishing write tid 1 to smithi15938319-3 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1961: finishing write tid 2 to smithi15938319-3 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1961: finishing write tid 3 to smithi15938319-3 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1961: finishing write tid 4 to smithi15938319-3 2024-09-13T13:52:29.186 INFO:tasks.rados.rados.0.smithi159.stdout:1961: oid 3 updating version 0 to 366 2024-09-13T13:52:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:1961: oid 3 updating version 366 to 368 2024-09-13T13:52:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:1961: oid 3 version 368 is already newer than 367 2024-09-13T13:52:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 368 (ObjNum 384 snap 60 seq_num 384) dirty exists 2024-09-13T13:52:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:1961: left oid 3 (ObjNum 384 snap 60 seq_num 384) 2024-09-13T13:52:29.187 INFO:tasks.rados.rados.0.smithi159.stdout:1962: expect (ObjNum 295 snap 41 seq_num 295) 2024-09-13T13:52:29.198 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:29.197+0000 7fc16a180640 -1 osd.11 163 handle_osd_map injecting map crc failure 2024-09-13T13:52:29.198 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:29.198+0000 7f9b0c553640 -1 osd.12 164 handle_osd_map injecting map crc failure 2024-09-13T13:52:29.221 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:29.221+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:29.258 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 494014 2024-09-13T13:52:29.287 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.287 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.296 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:29.299 INFO:tasks.rados.rados.0.smithi159.stdout:1963: finishing write tid 1 to smithi15938319-15 2024-09-13T13:52:29.299 INFO:tasks.rados.rados.0.smithi159.stdout:1963: finishing write tid 2 to smithi15938319-15 2024-09-13T13:52:29.299 INFO:tasks.rados.rados.0.smithi159.stdout:1963: finishing write tid 3 to smithi15938319-15 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:1963: finishing write tid 4 to smithi15938319-15 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:1963: oid 15 updating version 0 to 350 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:1963: oid 15 updating version 350 to 352 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:1963: oid 15 version 352 is already newer than 351 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 15 v 352 (ObjNum 385 snap 60 seq_num 385) dirty exists 2024-09-13T13:52:29.300 INFO:tasks.rados.rados.0.smithi159.stdout:1963: left oid 15 (ObjNum 385 snap 60 seq_num 385) 2024-09-13T13:52:29.301 INFO:tasks.rados.rados.0.smithi159.stdout:1964: finishing write tid 1 to smithi15938319-41 2024-09-13T13:52:29.301 INFO:tasks.rados.rados.0.smithi159.stdout:1964: finishing write tid 2 to smithi15938319-41 2024-09-13T13:52:29.301 INFO:tasks.rados.rados.0.smithi159.stdout:1964: finishing write tid 3 to smithi15938319-41 2024-09-13T13:52:29.301 INFO:tasks.rados.rados.0.smithi159.stdout:1964: finishing write tid 4 to smithi15938319-41 2024-09-13T13:52:29.301 INFO:tasks.rados.rados.0.smithi159.stdout:1964: oid 41 updating version 0 to 354 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1964: oid 41 updating version 354 to 356 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1964: oid 41 version 356 is already newer than 355 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 41 v 356 (ObjNum 386 snap 60 seq_num 386) dirty exists 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1964: left oid 41 (ObjNum 386 snap 60 seq_num 386) 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1961: done (4 left) 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1962: done (3 left) 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1963: done (2 left) 2024-09-13T13:52:29.302 INFO:tasks.rados.rados.0.smithi159.stdout:1964: done (1 left) 2024-09-13T13:52:29.303 INFO:tasks.rados.rados.0.smithi159.stdout:1967: delete oid 48 current snap is 60 2024-09-13T13:52:29.303 INFO:tasks.rados.rados.0.smithi159.stdout:1966: got (22) Invalid argument 2024-09-13T13:52:29.306 INFO:tasks.rados.rados.0.smithi159.stdout:1966: done (1 left) 2024-09-13T13:52:29.307 INFO:tasks.rados.rados.0.smithi159.stdout:1967: done (0 left) 2024-09-13T13:52:29.307 INFO:tasks.rados.rados.0.smithi159.stdout:1968: write oid 12 current snap is 60 2024-09-13T13:52:29.307 INFO:tasks.rados.rados.0.smithi159.stdout:1968: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-13T13:52:29.313 INFO:tasks.rados.rados.0.smithi159.stdout:1968: writing smithi15938319-12 from 521116 to 1251731 tid 1 2024-09-13T13:52:29.320 INFO:tasks.rados.rados.0.smithi159.stdout:1968: writing smithi15938319-12 from 1971484 to 2521116 tid 2 2024-09-13T13:52:29.320 INFO:tasks.rados.rados.0.smithi159.stdout:1969: snap_create 2024-09-13T13:52:29.321 INFO:tasks.rados.rados.0.smithi159.stdout:1968: finishing write tid 1 to smithi15938319-12 2024-09-13T13:52:29.323 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:29.322+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:29.323 INFO:tasks.rados.rados.0.smithi159.stdout:1968: finishing write tid 2 to smithi15938319-12 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:1968: finishing write tid 3 to smithi15938319-12 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:1968: finishing write tid 4 to smithi15938319-12 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:1968: oid 12 updating version 0 to 363 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:1968: oid 12 version 363 is already newer than 362 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:1968: oid 12 updating version 363 to 364 2024-09-13T13:52:29.324 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 12 v 364 (ObjNum 388 snap 60 seq_num 388) dirty exists 2024-09-13T13:52:29.325 INFO:tasks.rados.rados.0.smithi159.stdout:1968: left oid 12 (ObjNum 388 snap 60 seq_num 388) 2024-09-13T13:52:29.407 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.408 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.416 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:29.424 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:29.423+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:29.525 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:29.525+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:29.560 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.560 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.568 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:29.626 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:29.625+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:29.671 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.672 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.680 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:29.727 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:29.727+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:29.825 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.825 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.828 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:29.828+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:29.834 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:29.930 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:29.929+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:29.976 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:29.976 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:29.985 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:30.030 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:30.030+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:30.088 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.088 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:30.131 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:30.131+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:30.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:30.189 INFO:tasks.rados.rados.0.smithi159.stdout:1968: done (1 left) 2024-09-13T13:52:30.189 INFO:tasks.rados.rados.0.smithi159.stdout:1969: done (0 left) 2024-09-13T13:52:30.189 INFO:tasks.rados.rados.0.smithi159.stdout:1970: read oid 39 snap -1 2024-09-13T13:52:30.189 INFO:tasks.rados.rados.0.smithi159.stdout:1970: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:30.189 INFO:tasks.rados.rados.0.smithi159.stdout:1971: set_chunk oid 13 target oid 2024-09-13T13:52:30.203 INFO:tasks.rados.rados.0.smithi159.stdout:1971: set_chunk oid 13 offset: 1466880 length: 126464 target oid faa507b34025dbc537476a12c2524b48a0c6632aba70094f7272f0ca404b42f4 offset: 0 snap -1 2024-09-13T13:52:30.203 INFO:tasks.rados.rados.0.smithi159.stdout:1972: read oid 2 snap 60 2024-09-13T13:52:30.203 INFO:tasks.rados.rados.0.smithi159.stdout:1972: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:30.203 INFO:tasks.rados.rados.0.smithi159.stdout:1973: read oid 33 snap 55 2024-09-13T13:52:30.203 INFO:tasks.rados.rados.0.smithi159.stdout:1973: expect (ObjNum 356 snap 51 seq_num 356) 2024-09-13T13:52:30.204 INFO:tasks.rados.rados.0.smithi159.stdout:1974: read oid 23 snap -1 2024-09-13T13:52:30.204 INFO:tasks.rados.rados.0.smithi159.stdout:1974: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:30.204 INFO:tasks.rados.rados.0.smithi159.stdout:1975: read oid 3 snap -1 2024-09-13T13:52:30.204 INFO:tasks.rados.rados.0.smithi159.stdout:1975: expect (ObjNum 384 snap 60 seq_num 384) 2024-09-13T13:52:30.204 INFO:tasks.rados.rados.0.smithi159.stdout:1976: snap_create 2024-09-13T13:52:30.205 INFO:tasks.rados.rados.0.smithi159.stdout:1970: expect (ObjNum 329 snap 47 seq_num 329) 2024-09-13T13:52:30.220 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.220 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:30.232 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:30.231+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:30.308 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:30.307+0000 7f4ffdd7e640 -1 osd.1 165 handle_osd_map injecting map crc failure 2024-09-13T13:52:30.329 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 317837 2024-09-13T13:52:30.333 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:30.332+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:30.383 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.383 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:30.396 INFO:tasks.rados.rados.0.smithi159.stdout:1971: finishing set_chunk to oid 13 2024-09-13T13:52:30.396 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 13 v 198 (ObjNum 372 snap 54 seq_num 372) dirty exists 2024-09-13T13:52:30.396 INFO:tasks.rados.rados.0.smithi159.stdout:1975: expect (ObjNum 384 snap 60 seq_num 384) 2024-09-13T13:52:30.433 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:30.433+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:30.500 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.500 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:30.514 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:30.514+0000 7ffaeece8640 -1 osd.0 165 handle_osd_map injecting map crc failure 2024-09-13T13:52:30.515 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:30.514+0000 7f2282f0c640 -1 osd.10 165 handle_osd_map injecting map crc failure 2024-09-13T13:52:30.528 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2360698 2024-09-13T13:52:30.534 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:30.534+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:30.617 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.617 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.617 INFO:tasks.rados.rados.0.smithi159.stdout:1973: expect (ObjNum 356 snap 51 seq_num 356) 2024-09-13T13:52:30.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:30.634 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:30.633+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:30.676 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1399892 2024-09-13T13:52:30.736 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:30.735+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:30.745 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2792554 2024-09-13T13:52:30.767 INFO:tasks.rados.rados.0.smithi159.stdout:1972: expect (ObjNum 318 snap 46 seq_num 318) 2024-09-13T13:52:30.767 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:30.772 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.773 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.781 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:30.837 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:30.836+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:30.847 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 438180 2024-09-13T13:52:30.919 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:30.928 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:30.929 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:30.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:30.939 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:30.938+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:30.958 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:30.996 INFO:tasks.rados.rados.0.smithi159.stdout:1974: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:31.023 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1820801 2024-09-13T13:52:31.039 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1154566 2024-09-13T13:52:31.040 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:31.039+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:31.045 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:31.045 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:31.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:31.117 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:31.117 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:31.140 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:31.141 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:31.140+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:31.162 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:31.183 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1970: done (6 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1971: done (5 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1972: done (4 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1973: done (3 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1974: done (2 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1975: done (1 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1976: done (0 left) 2024-09-13T13:52:31.197 INFO:tasks.rados.rados.0.smithi159.stdout:1977: snap_create 2024-09-13T13:52:31.206 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:31.206 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:31.208 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:31.209 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:31.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:31.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:31.230 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:31.241 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:31.241+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:31.249 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:31.248+0000 7f7c71c1d640 -1 osd.14 166 handle_osd_map injecting map crc failure 2024-09-13T13:52:31.250 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:31.248+0000 7fc16a180640 -1 osd.11 166 handle_osd_map injecting map crc failure 2024-09-13T13:52:31.255 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:31.255 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:31.276 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:31.297 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:31.298 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:31.319 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:31.338 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:31.338 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:31.341 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:31.342 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:31.341+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:31.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:31.361 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:31.383 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:31.383 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:31.405 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:31.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:31.405+0000 7fb3ba59c640 0 -- 172.21.15.27:0/3245668818 >> v2:172.21.15.159:6805/4073091144 conn(0x7fb3bc2222d0 msgr2=0x7fb3bc2246c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:31.443 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:31.442+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:31.463 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:31.463 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:31.463 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:31.463 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:31.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:31.543 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:31.543+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:31.576 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:31.577 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:31.577 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:31.577 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:31.577 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:31.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:31.628 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:31.632 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:31.644 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:31.643+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 138.773049375, 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:52:31.691 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:31.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:52:31.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:31.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 138.768249846, 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:52:31.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:52:31.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:31.705 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 137.91890180799999, 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:31.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:52:31.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.414604208, 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.713 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:52:31.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:52:31.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:31.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:52:31.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:52:31.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.408744259, 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.722 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:52:31.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:52:31.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:52:31.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.40379961799999, 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.731 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:52:31.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:52:31.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:52:31.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:52:31.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.739 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.40358904, 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.740 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:31.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:52:31.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:31.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:52:31.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:31.748 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.403540941, 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:52:31.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:52:31.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:52:31.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.32046053000001, 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:52:31.756 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:52:31.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:52:31.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.762 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.320351819, 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.763 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:31.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:52:31.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:52:31.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:52:31.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:52:31.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:52:31.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:52:31.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:52:31.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:52:31.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:52:31.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:52:31.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:52:31.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.775 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:31.776 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:31.776 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.32023653900001, 2024-09-13T13:52:31.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:52:31.776 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.776 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:52:31.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:52:31.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:52:31.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:52:31.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:52:31.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:52:31.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:52:31.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:52:31.786 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 89.812266605000005, 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:52:31.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:52:31.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.793 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(osd.1.0:130 4.7 e130/52 ondisk, result = 0)", 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 39.874049165999999, 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18760375600000001, 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814292+0000", 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7999999999999995e-07 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814296+0000", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8339999999999998e-06 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001621+0000", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18732465400000001 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001634+0000", 2024-09-13T13:52:31.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2883e-05 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001774+0000", 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000140104 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001805+0000", 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001895+0000", 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9456000000000005e-05 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4040 4.7 4:e26a40dd:::smithi15938319-25:head [write 763117~710598] snapc 30=[30,2f,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 30.560321319, 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010818598, 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.798 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4040 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128020+0000", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.778e-06 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128710+0000", 2024-09-13T13:52:31.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000689637 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128727+0000", 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6628000000000001e-05 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128734+0000", 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128785+0000", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0828999999999997e-05 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128789+0000", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1409999999999998e-06 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133000+0000", 2024-09-13T13:52:31.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042116080000000004 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133067+0000", 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6644999999999996e-05 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133188+0000", 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121249 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138729+0000", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055408610000000002 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138749+0000", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0029000000000001e-05 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138796+0000", 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6681999999999998e-05 2024-09-13T13:52:31.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138837+0000", 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1440000000000003e-05 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4225 4.7 4:e26a40dd:::smithi15938319-25:23 [read 0~3630279,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.363589077, 2024-09-13T13:52:31.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010120242, 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4225 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:31.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8000000000000001e-07 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324754+0000", 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6419999999999999e-06 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324758+0000", 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6629999999999999e-06 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324761+0000", 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.145e-06 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324877+0000", 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116599 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324884+0000", 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4520000000000002e-06 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329877+0000", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049932090000000002 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329973+0000", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5606000000000006e-05 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334871+0000", 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048985460000000002 2024-09-13T13:52:31.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4226 4.7 4:e26a40dd:::smithi15938319-25:23 [sparse-read 844183~2211810] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 24.363557802999999, 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011467577, 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.811 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4226 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324783+0000", 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324784+0000", 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.731e-06 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324789+0000", 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324796+0000", 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.537e-06 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324912+0000", 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115145 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324918+0000", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6100000000000002e-06 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334923+0000", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010004569 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334990+0000", 2024-09-13T13:52:31.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7143000000000004e-05 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.336250+0000", 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012599320000000001 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4417 4.7 4:e26a40dd:::smithi15938319-25:head [write 484134~708729,stat] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:31.816 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.772871313, 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013723770999999999, 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4417 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915473+0000", 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.827e-06 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916562+0000", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089575 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916582+0000", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9825000000000001e-05 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916590+0000", 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.038e-06 2024-09-13T13:52:31.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916652+0000", 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1941999999999997e-05 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916657+0000", 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921371+0000", 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047142629999999998 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921469+0000", 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7625000000000001e-05 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921620+0000", 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015109999999999999 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928797+0000", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071769199999999998 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929103+0000", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030604200000000002 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929152+0000", 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8565e-05 2024-09-13T13:52:31.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929192+0000", 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0849000000000001e-05 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4418 4.7 4:e26a40dd:::smithi15938319-25:head [write 1883658~200476] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.768060042, 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014278255, 2024-09-13T13:52:31.823 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4418 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:31.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920281+0000", 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.24e-06 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920579+0000", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029755100000000001 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920594+0000", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4951e-05 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920601+0000", 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8950000000000001e-06 2024-09-13T13:52:31.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920658+0000", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7017e-05 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920666+0000", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4370000000000007e-06 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924237+0000", 2024-09-13T13:52:31.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035709560000000001 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924325+0000", 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8212000000000002e-05 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924457+0000", 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000131686 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929201+0000", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047441380000000002 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934475+0000", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052734549999999998 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934520+0000", 2024-09-13T13:52:31.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5423999999999997e-05 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934558+0000", 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8293000000000001e-05 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4419 4.7 4:e26a40dd:::smithi15938319-25:head [setxattr _header (57) in=7b,truncate 2084134] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:31.830 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.767723589999999, 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014942327, 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4419 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920617+0000", 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1800000000000005e-07 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920621+0000", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9559999999999999e-06 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920630+0000", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3200000000000006e-06 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920637+0000", 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2129999999999996e-06 2024-09-13T13:52:31.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920708+0000", 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1546999999999998e-05 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920715+0000", 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3579999999999999e-06 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.927744+0000", 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070291900000000003 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.927853+0000", 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109227 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928066+0000", 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021321400000000001 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.933043+0000", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049770439999999999 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935446+0000", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002402894 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935481+0000", 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4431000000000003e-05 2024-09-13T13:52:31.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935559+0000", 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8214999999999996e-05 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4420 4.7 4:e26a40dd:::smithi15938319-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.767688838, 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01508695, 2024-09-13T13:52:31.837 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4420 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920652+0000", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8400000000000004e-07 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920655+0000", 2024-09-13T13:52:31.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0469999999999998e-06 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920661+0000", 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920667+0000", 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1880000000000001e-06 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920742+0000", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5790000000000005e-05 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920746+0000", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4819999999999999e-06 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928833+0000", 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0080870690000000005 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928895+0000", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2094999999999994e-05 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935599+0000", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00670383 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935659+0000", 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9737000000000001e-05 2024-09-13T13:52:31.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935738+0000", 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9555000000000007e-05 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:31.843 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:31.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:31.845 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:31.844+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:31.945 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:31.945+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:31.948 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:31.949 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:31.949 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:31.949 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:31.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:32.047 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:32.046+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:32.100 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.100 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:32.100 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:32.100 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:32.100 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:32.125 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:32.124+0000 7fe0f9e03640 -1 osd.2 166 handle_osd_map injecting map crc failure 2024-09-13T13:52:32.148 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:32.147+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:32.196 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:32.195+0000 7f002fbdf640 -1 osd.15 167 handle_osd_map injecting map crc failure 2024-09-13T13:52:32.197 INFO:tasks.rados.rados.0.smithi159.stdout:1977: done (0 left) 2024-09-13T13:52:32.197 INFO:tasks.rados.rados.0.smithi159.stdout:1978: delete oid 45 current snap is 63 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1978: done (0 left) 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1979: read oid 10 snap -1 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1979: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1980: rollback oid 8 current snap is 63 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 8 to 62 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1981: write oid 44 current snap is 63 2024-09-13T13:52:32.207 INFO:tasks.rados.rados.0.smithi159.stdout:1981: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-13T13:52:32.213 INFO:tasks.rados.rados.0.smithi159.stdout:1981: writing smithi15938319-44 from 554730 to 1035843 tid 1 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 138.44289911800001, 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:52:32.214 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.215 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.215 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:32.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:52:32.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:52:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:32.221 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 138.43386868600001, 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:32.226 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.78186281000001, 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:52:32.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 133.77882882399999, 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:32.231 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:52:32.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 116.902996706, 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.235 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:52:32.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:52:32.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 116.897089874, 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:32.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:52:32.244 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 116.889654439, 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:52:32.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.970360697, 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.249 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:52:32.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:32.255 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.970292032, 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:52:32.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:52:32.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.970241918, 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:52:32.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:52:32.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.949284557, 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:52:32.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:52:32.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:52:32.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.943688354, 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:52:32.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:52:32.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.94214444399999, 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:52:32.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:32.287 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.942079396, 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:52:32.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1862 4.4 4:21c00763:::smithi15938319-21:head [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.335137797, 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066286849999999996, 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.293 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1862 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877196+0000", 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877197+0000", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6499999999999999e-07 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877201+0000", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6929999999999998e-06 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877207+0000", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.877212+0000", 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8489999999999997e-06 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881042+0000", 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038299969999999999 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.881156+0000", 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114267 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.883825+0000", 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026686729999999999 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 101.479462094, 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:52:32.297 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:52:32.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 101.407566406, 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.303 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:52:32.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:52:32.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 101.407528885, 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:52:32.309 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:52:32.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:52:32.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.29461013, 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.315 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:32.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:52:32.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:52:32.321 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 96.080543722000002, 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:32.322 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:52:32.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:32.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.328 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.329 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:32.232+0000 7f2284f10640 -1 osd.10 167 handle_osd_map injecting map crc failure 2024-09-13T13:52:32.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:32.331 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:32.329+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1981: writing smithi15938319-44 from 1572095 to 2309549 tid 2 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1982: set_chunk oid 9 target oid 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1982: set_chunk oid 9 offset: 427520 length: 1619456 target oid af358a505bfa0624c159c119e497d73f87b80dafc3fee28f2b214096cf345d9b offset: 0 snap -1 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1983: set_chunk oid 25 target oid 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1983: set_chunk oid 25 offset: 531968 length: 512 target oid f36a7a42222bc329d00942315cee61626a2cba5582dda3b7300fa6a25b0c67e6 offset: 0 snap -1 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1984: read oid 43 snap -1 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1984: expect deleted 2024-09-13T13:52:32.332 INFO:tasks.rados.rados.0.smithi159.stdout:1985: read oid 29 snap 50 2024-09-13T13:52:32.333 INFO:tasks.rados.rados.0.smithi159.stdout:1985: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:32.333 INFO:tasks.rados.rados.0.smithi159.stdout:1986: read oid 40 snap 32 2024-09-13T13:52:32.333 INFO:tasks.rados.rados.0.smithi159.stdout:1986: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:32.333 INFO:tasks.rados.rados.0.smithi159.stdout:1987: set_chunk oid 27 target oid 2024-09-13T13:52:32.333 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:32.265+0000 7fe857f4f640 -1 osd.7 167 handle_osd_map injecting map crc failure 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:1987: set_chunk oid 27 offset: 279040 length: 446976 target oid a822e7508bb32ea5dd57b0030e9943977b137986d68721c435f0acaebc41706c offset: 0 snap 50 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:1988: tier_flush oid 26 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:1988: tier_flush oid 26 snap -1 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:1979: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:32.334 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1980: finishing rollback tid 0 to smithi15938319-8 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1980: finishing rollback tid 1 to smithi15938319-8 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1979: done (9 left) 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1989: read oid 50 snap 61 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1989: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1990: read oid 17 snap -1 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1990: expect (ObjNum 369 snap 54 seq_num 369) 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1991: tier_promote oid 32 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:1980: finishing rollback tid 2 to smithi15938319-8 2024-09-13T13:52:32.409 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 245 (ObjNum 352 snap 50 seq_num 352) dirty exists 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: finishing write tid 1 to smithi15938319-44 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: finishing write tid 2 to smithi15938319-44 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: finishing write tid 3 to smithi15938319-44 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: finishing write tid 4 to smithi15938319-44 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: oid 44 updating version 0 to 366 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: oid 44 updating version 366 to 367 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: oid 44 updating version 367 to 368 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 44 v 368 (ObjNum 390 snap 63 seq_num 390) dirty exists 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1981: left oid 44 (ObjNum 390 snap 63 seq_num 390) 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1982: finishing set_chunk to oid 9 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 427520 length: 1619456 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 0 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:52:32.410 INFO:tasks.rados.rados.0.smithi159.stdout:1983: finishing set_chunk to oid 25 2024-09-13T13:52:32.411 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 25 v 253 (ObjNum 362 snap 52 seq_num 362) dirty exists 2024-09-13T13:52:32.411 INFO:tasks.rados.rados.0.smithi159.stdout:1985: expect (ObjNum 87 snap 5 seq_num 87) 2024-09-13T13:52:32.430 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:32.429+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:32.470 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1875988 2024-09-13T13:52:32.474 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.474 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:32.474 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:32.474 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.483 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:32.491 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:32.531 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:32.531+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:32.538 INFO:tasks.rados.rados.0.smithi159.stdout:1986: expect (ObjNum 175 snap 16 seq_num 175) 2024-09-13T13:52:32.585 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.586 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:32.586 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:32.586 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:32.586 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:32.632 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:32.631+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:32.666 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2127708 2024-09-13T13:52:32.682 INFO:tasks.rados.rados.0.smithi159.stdout:1987: finishing set_chunk to oid 27 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 279040 length: 446976 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 27 v 0 (ObjNum 377 snap 57 seq_num 377) dirty exists 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1988: got (16) Device or resource busy 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:check_oldest_snap_flushed oid 26 v 0 (ObjNum 49 snap 0 seq_num 1515034256) unflushed 52 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1980: done (11 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1981: done (10 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1982: done (9 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1983: done (8 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1984: done (7 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1985: done (6 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1986: done (5 left) 2024-09-13T13:52:32.683 INFO:tasks.rados.rados.0.smithi159.stdout:1987: done (4 left) 2024-09-13T13:52:32.684 INFO:tasks.rados.rados.0.smithi159.stdout:1988: done (3 left) 2024-09-13T13:52:32.684 INFO:tasks.rados.rados.0.smithi159.stdout:1992: write oid 19 current snap is 63 2024-09-13T13:52:32.684 INFO:tasks.rados.rados.0.smithi159.stdout:1992: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-13T13:52:32.688 INFO:tasks.rados.rados.0.smithi159.stdout:1992: writing smithi15938319-19 from 571537 to 1127899 tid 1 2024-09-13T13:52:32.694 INFO:tasks.rados.rados.0.smithi159.stdout:1992: writing smithi15938319-19 from 1630577 to 2228042 tid 2 2024-09-13T13:52:32.694 INFO:tasks.rados.rados.0.smithi159.stdout:1993: read oid 14 snap -1 2024-09-13T13:52:32.694 INFO:tasks.rados.rados.0.smithi159.stdout:1993: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:32.695 INFO:tasks.rados.rados.0.smithi159.stdout:1994: tier_promote oid 43 2024-09-13T13:52:32.695 INFO:tasks.rados.rados.0.smithi159.stdout:1991: got (0) Success 2024-09-13T13:52:32.695 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 32 v 206 (ObjNum 220 snap 24 seq_num 220) dirty exists 2024-09-13T13:52:32.695 INFO:tasks.rados.rados.0.smithi159.stdout:1990: expect (ObjNum 369 snap 54 seq_num 369) 2024-09-13T13:52:32.698 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.698 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:32.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:32.698 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 161.463289379, 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088411000000000002, 2024-09-13T13:52:32.699 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:32.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234557+0000", 2024-09-13T13:52:32.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:32.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234564+0000", 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2160000000000003e-06 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234603+0000", 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9674e-05 2024-09-13T13:52:32.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234616+0000", 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2473999999999999e-05 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234701+0000", 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5358999999999994e-05 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.234729+0000", 2024-09-13T13:52:32.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7644000000000002e-05 2024-09-13T13:52:32.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235376+0000", 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064700999999999995 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:51.235441+0000", 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5563999999999998e-05 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4290.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 160.455255233, 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072733599999999998, 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:32.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242591+0000", 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242592+0000", 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.28e-07 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242597+0000", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4759999999999997e-06 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242623+0000", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6140999999999999e-05 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242634+0000", 2024-09-13T13:52:32.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1248e-05 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242718+0000", 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3617999999999998e-05 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.242734+0000", 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.59e-05 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243293+0000", 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055931000000000004 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:49:52.243319+0000", 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5415e-05 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.714 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(2.2 64/63 [PushReplyOp(2:5cea7035:::rbd_info:head)])", 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 111.059557074, 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022591399999999999, 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638289+0000", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638290+0000", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.98e-07 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638293+0000", 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-13T13:52:32.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638327+0000", 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3877000000000001e-05 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638335+0000", 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1249999999999993e-06 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638377+0000", 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2178000000000002e-05 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.638515+0000", 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000138026 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,QREQUEST,e:96,from:1,reservation_nonce:2)", 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:32.718 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 79.512653372000003, 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004235094, 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185193+0000", 2024-09-13T13:52:32.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185194+0000", 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1939999999999999e-06 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185202+0000", 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4420000000000001e-06 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185238+0000", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6236999999999999e-05 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185246+0000", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5059999999999998e-06 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185290+0000", 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4802999999999998e-05 2024-09-13T13:52:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.185295+0000", 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.566e-06 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189388+0000", 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040932820000000002 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189405+0000", 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6552e-05 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.189428+0000", 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3512000000000001e-05 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "replica_scrub(pg: 2.4,from:0'0,to:0'0,epoch:96/25,start:2:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-13T13:52:32.723 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 79.507812548000004, 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115704, 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190034+0000", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190035+0000", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.333e-06 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190042+0000", 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7429999999999996e-06 2024-09-13T13:52:32.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190076+0000", 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4258999999999998e-05 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190083+0000", 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190122+0000", 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8933999999999997e-05 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190128+0000", 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190150+0000", 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1382e-05 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDScrubReserve(2.4,RELEASE,e:96,from:1,reservation_nonce:0)", 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 79.507099573000005, 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000185336, 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:32.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190747+0000", 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.91e-07 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190750+0000", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3180000000000001e-06 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190802+0000", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2224e-05 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190808+0000", 2024-09-13T13:52:32.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0730000000000001e-06 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190896+0000", 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7899999999999995e-05 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190910+0000", 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4112e-05 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:32.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:13.190932+0000", 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2418000000000001e-05 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:32.732 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:32.734 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:32.733+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:32.834 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:32.834+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:32.848 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1850822 2024-09-13T13:52:32.851 INFO:tasks.rados.rados.0.smithi159.stdout:1989: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:32.878 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:32.878 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:32.878 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:32.878 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:32.886 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:32.910 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:32.935 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:32.935+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:32.992 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3239792 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1989: done (5 left) 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1990: done (4 left) 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1991: done (3 left) 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1995: read oid 23 snap -1 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1995: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1992: finishing write tid 1 to smithi15938319-19 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1994: got (0) Success 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 43 v 239 (ObjNum 49 snap 0 seq_num 1515034256) clean dne 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1992: finishing write tid 2 to smithi15938319-19 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1996: read oid 32 snap -1 2024-09-13T13:52:33.007 INFO:tasks.rados.rados.0.smithi159.stdout:1996: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1994: done (4 left) 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1997: read oid 41 snap -1 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1997: expect (ObjNum 386 snap 60 seq_num 386) 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1998: read oid 35 snap -1 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1998: expect deleted 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1999: read oid 22 snap -1 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1999: expect deleted 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:2000: snap_create 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1992: finishing write tid 3 to smithi15938319-19 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1992: finishing write tid 4 to smithi15938319-19 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1992: oid 19 updating version 0 to 272 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1992: oid 19 version 272 is already newer than 271 2024-09-13T13:52:33.008 INFO:tasks.rados.rados.0.smithi159.stdout:1992: oid 19 updating version 272 to 273 2024-09-13T13:52:33.009 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 19 v 273 (ObjNum 391 snap 63 seq_num 391) dirty exists 2024-09-13T13:52:33.009 INFO:tasks.rados.rados.0.smithi159.stdout:1992: left oid 19 (ObjNum 391 snap 63 seq_num 391) 2024-09-13T13:52:33.009 INFO:tasks.rados.rados.0.smithi159.stdout:1993: expect (ObjNum 315 snap 45 seq_num 315) 2024-09-13T13:52:33.030 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:33.031 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:33.031 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:33.031 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:33.031 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:33.033 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1811089 2024-09-13T13:52:33.036 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:33.035+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:33.039 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:33.134 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:33.137 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:33.136+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:217 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.376075907, 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.031318771000000002, 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.145 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 217 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766803+0000", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766805+0000", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.054e-06 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766812+0000", 2024-09-13T13:52:33.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3170000000000004e-06 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766824+0000", 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.216e-05 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766830+0000", 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6300000000000003e-06 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.766887+0000", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7775000000000002e-05 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767015+0000", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012774800000000001 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.783855+0000", 2024-09-13T13:52:33.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016839811 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790317+0000", 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064624069999999999 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798051+0000", 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077339610000000001 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798085+0000", 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3980000000000003e-05 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798122+0000", 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6928000000000003e-05 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:220 3.1 3:88892e40:::smithi15938319-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.151 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.37509870900001, 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.052488110999999997, 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 220 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767781+0000", 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2900000000000003e-07 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767785+0000", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767792+0000", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6720000000000004e-06 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767797+0000", 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.481e-06 2024-09-13T13:52:33.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767836+0000", 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8794000000000002e-05 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.767917+0000", 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1729000000000003e-05 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.804005+0000", 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03608803 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809423+0000", 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054175229999999996 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820202+0000", 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010778879999999999 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820237+0000", 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5272999999999999e-05 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.820269+0000", 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1780999999999999e-05 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.157 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:223 3.1 3:932c05fb:::smithi15938319-24 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.37334404399999, 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028733892, 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 223 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.158 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769535+0000", 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769536+0000", 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769539+0000", 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769546+0000", 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8190000000000003e-06 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769550+0000", 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4429999999999996e-06 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769589+0000", 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8886e-05 2024-09-13T13:52:33.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769685+0000", 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5971000000000007e-05 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.787107+0000", 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017421803 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.793874+0000", 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067664539999999999 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798172+0000", 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042980450000000003 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798221+0000", 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9906e-05 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798269+0000", 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7750000000000002e-05 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.164 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:224 3.1 3:9203634f:::smithi15938319-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.37295183099999, 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020380514999999998, 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 224 2024-09-13T13:52:33.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769928+0000", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769943+0000", 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0577999999999999e-05 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769951+0000", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7589999999999997e-06 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769980+0000", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9666e-05 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770059+0000", 2024-09-13T13:52:33.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8757000000000005e-05 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.779634+0000", 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009575152 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790174+0000", 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010540208000000001 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790200+0000", 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5194e-05 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790255+0000", 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5553000000000002e-05 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790308+0000", 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2880000000000002e-05 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:225 3.1 3:b64df3d0:::smithi15938319-26 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.37291450399999, 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028375300999999999, 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.171 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 225 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769965+0000", 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6899999999999998e-07 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769967+0000", 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.114e-06 2024-09-13T13:52:33.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769974+0000", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2079999999999997e-06 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.769978+0000", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770141+0000", 2024-09-13T13:52:33.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016302999999999999 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.770227+0000", 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5679000000000004e-05 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.788744+0000", 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.018517250999999998 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.794187+0000", 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054426359999999998 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798294+0000", 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041073409999999996 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798319+0000", 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5236999999999999e-05 2024-09-13T13:52:33.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798340+0000", 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0988e-05 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:228 3.1 3:81f902ac:::smithi15938319-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:33.177 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.36811615100001, 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.034837912999999998, 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 228 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774763+0000", 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774764+0000", 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7600000000000006e-07 2024-09-13T13:52:33.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774778+0000", 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3850000000000008e-06 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774791+0000", 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2306999999999999e-05 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774828+0000", 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7898999999999997e-05 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8258999999999997e-05 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.795897+0000", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020989985999999999 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798352+0000", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00245575 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809467+0000", 2024-09-13T13:52:33.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011114251 2024-09-13T13:52:33.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809541+0000", 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4128999999999994e-05 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.809601+0000", 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0300999999999997e-05 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:231 3.1 3:9f3f9b00:::smithi15938319-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.352480926, 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032497671999999998, 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.185 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 231 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790398+0000", 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790400+0000", 2024-09-13T13:52:33.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-13T13:52:33.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790406+0000", 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4099999999999996e-06 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790417+0000", 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1192999999999999e-05 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790428+0000", 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0665e-05 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790478+0000", 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9892999999999998e-05 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790571+0000", 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3263999999999993e-05 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.808010+0000", 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017438393 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.812635+0000", 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046250270000000003 2024-09-13T13:52:33.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822824+0000", 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010189327 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822860+0000", 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5666e-05 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822896+0000", 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6414000000000001e-05 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:232 3.1 3:bba7b70d:::smithi15938319-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.35237457599999, 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032490036999999999, 2024-09-13T13:52:33.192 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 232 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790505+0000", 2024-09-13T13:52:33.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3799999999999997e-07 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790509+0000", 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5590000000000001e-06 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790514+0000", 2024-09-13T13:52:33.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3210000000000004e-06 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790518+0000", 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.737e-06 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790697+0000", 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000178591 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.790806+0000", 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109613 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.810761+0000", 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01995444 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.813915+0000", 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031546650000000001 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822926+0000", 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090111529999999992 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822972+0000", 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5779000000000003e-05 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.822995+0000", 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2640999999999999e-05 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:235 3.1 3:9761ca94:::smithi15938319-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.199 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.34444080599999, 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.028255987999999999, 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 235 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798439+0000", 2024-09-13T13:52:33.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9899999999999999e-07 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798444+0000", 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5850000000000001e-06 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798459+0000", 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4965e-05 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798464+0000", 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1440000000000002e-06 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798496+0000", 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2410000000000003e-05 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.798581+0000", 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4078000000000004e-05 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.815378+0000", 2024-09-13T13:52:33.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016797333000000001 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817724+0000", 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023456330000000002 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826628+0000", 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0089049479999999993 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826662+0000", 2024-09-13T13:52:33.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3800000000000002e-05 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.826695+0000", 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2292999999999997e-05 2024-09-13T13:52:33.208 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:242 3.1 3:835efa6b:::smithi15938319-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.209 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.326052248, 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023273500999999999, 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.210 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 242 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:33.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816827+0000", 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816828+0000", 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:33.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816832+0000", 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4229999999999998e-06 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816839+0000", 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7870000000000004e-06 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816844+0000", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3890000000000004e-06 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817083+0000", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023876999999999999 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817223+0000", 2024-09-13T13:52:33.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014026500000000001 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.831365+0000", 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014141341999999999 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.836182+0000", 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048173929999999997 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840021+0000", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038389629999999999 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840064+0000", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3148000000000001e-05 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840101+0000", 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6532999999999997e-05 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.216 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:244 3.1 3:b9de77c9:::smithi15938319-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.325925458, 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.027833673, 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 244 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:33.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816954+0000", 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8999999999999997e-07 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816957+0000", 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.836e-06 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816962+0000", 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4320000000000001e-06 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.816967+0000", 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8509999999999998e-06 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817302+0000", 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033517399999999999 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.817377+0000", 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5446999999999998e-05 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.838046+0000", 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020668659999999998 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.840793+0000", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002746718 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844646+0000", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0038531020000000002 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844736+0000", 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0434999999999997e-05 2024-09-13T13:52:33.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.844788+0000", 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1527999999999999e-05 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:248 3.1 3:960703b4:::smithi15938319-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.24472703500001, 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.022178494, 2024-09-13T13:52:33.223 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 248 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898152+0000", 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0099999999999998e-06 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0693000000000001e-05 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898192+0000", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6726000000000001e-05 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898209+0000", 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7139999999999999e-05 2024-09-13T13:52:33.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898362+0000", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000152946 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898530+0000", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000167801 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.906990+0000", 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084606439999999998 2024-09-13T13:52:33.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.911991+0000", 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050003089999999997 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920174+0000", 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081838090000000002 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920288+0000", 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011373400000000001 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920331+0000", 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2682000000000002e-05 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:260 4.f 4:fec51f4e:::smithi15938319-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:33.229 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.244625584, 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.017338078999999999, 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 260 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898254+0000", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898255+0000", 2024-09-13T13:52:33.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8700000000000004e-07 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898260+0000", 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2830000000000001e-06 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898274+0000", 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3675999999999999e-05 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898284+0000", 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9869999999999994e-06 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898428+0000", 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144239 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.898534+0000", 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105766 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912263+0000", 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0137289 2024-09-13T13:52:33.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.912787+0000", 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052415600000000001 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915506+0000", 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027194039999999999 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915545+0000", 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9082000000000003e-05 2024-09-13T13:52:33.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915592+0000", 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6699000000000001e-05 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1109 4.f 4:f2f09040:::smithi15938319-31:head [append 0~0] snapc 4=[3] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:33.237 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 128.68327281800001, 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46585949799999998, 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.238 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1109 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459607+0000", 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4499999999999998e-07 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459608+0000", 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.342e-06 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459611+0000", 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9220000000000001e-06 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.459614+0000", 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-13T13:52:33.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464525+0000", 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049109119999999999 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464616+0000", 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1073e-05 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.464796+0000", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017990599999999999 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465347+0000", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055148999999999997 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925377+0000", 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.460029939 2024-09-13T13:52:33.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925425+0000", 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8089e-05 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925466+0000", 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0772000000000002e-05 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:1109 4.f e48/34 ondisk, result = 0)", 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 128.677421275, 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.46001237499999997, 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465458+0000", 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465459+0000", 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.465462+0000", 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3189999999999999e-06 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925196+0000", 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.45973412200000002 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925227+0000", 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0982999999999998e-05 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925363+0000", 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013563900000000001 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925371+0000", 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6289999999999999e-06 2024-09-13T13:52:33.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:24.925470+0000", 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9179999999999996e-05 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.249 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1512 4.f 4:f2f09040:::smithi15938319-31:head [write 598349~480585] snapc 9=[9,8,7,6,5] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 119.415543824, 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016718751, 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1512 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727336+0000", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727337+0000", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.548e-06 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727861+0000", 2024-09-13T13:52:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052401699999999999 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727881+0000", 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0203999999999998e-05 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727893+0000", 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1584e-05 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.727995+0000", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010220799999999999 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.728003+0000", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4250000000000001e-06 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737026+0000", 2024-09-13T13:52:33.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090235999999999997 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737088+0000", 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2015999999999997e-05 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.737212+0000", 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000123709 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.740364+0000", 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003151905 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.743948+0000", 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035843630000000001 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744004+0000", 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6317000000000003e-05 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:33.744054+0000", 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9855000000000003e-05 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.4.23:16 3.1 3:90dfe591:::smithi15938319-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 118.409522549, 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.244064791, 2024-09-13T13:52:33.257 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733357+0000", 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0800000000000001e-07 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.733359+0000", 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5510000000000001e-06 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975364+0000", 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.242005198 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975373+0000", 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7139999999999999e-06 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975472+0000", 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9102000000000001e-05 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975572+0000", 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9992000000000001e-05 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.975742+0000", 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000170197 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.976598+0000", 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00085651399999999995 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977338+0000", 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074014600000000001 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977378+0000", 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9892999999999999e-05 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:34.977422+0000", 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3275999999999997e-05 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.263 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.7.23:20 3.1 3:8dbd923e:::ae8b86d07d375a083683440b6bda0e0f67b5195d585764d7ac407424ac114e59:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e122)", 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.126087376000001, 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89571672999999996, 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016792+0000", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016793+0000", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6600000000000005e-07 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.016800+0000", 2024-09-13T13:52:33.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1199999999999996e-06 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910876+0000", 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.89407579400000003 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910884+0000", 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.910982+0000", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7763999999999994e-05 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911101+0000", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119942 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.911338+0000", 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023690000000000001 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912020+0000", 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068117299999999998 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912443+0000", 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042320999999999998 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912478+0000", 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4768999999999999e-05 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:42.912509+0000", 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1226999999999998e-05 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "MOSDPGPushReply(3.1 126/125 [PushReplyOp(3:ac2af9f5:::e6b4f9f8265bfe5a786cb2bd77c53a79b3e484041c7c0186a813c80a2460efe6:head)])", 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:33.269 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 47.096668565000002, 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208631544, 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:33.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046211+0000", 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9399999999999995e-07 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.046216+0000", 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3959999999999999e-06 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254500+0000", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.208283991 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254505+0000", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.789e-06 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254594+0000", 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8101000000000002e-05 2024-09-13T13:52:33.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:46.254842+0000", 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024877299999999998 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(osd.6.22:64 3.1 3:92340f89:::0d01361f24984ff4b0b1eb9d9ead6642f5b878662a350c2f914a3a82940a2cbd:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e148)", 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.803796599000002, 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.044891857, 2024-09-13T13:52:33.273 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339083+0000", 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:33.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339084+0000", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.251e-06 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.339090+0000", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382252+0000", 2024-09-13T13:52:33.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043162303999999999 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382274+0000", 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1345999999999999e-05 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382467+0000", 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019345899999999999 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:33.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382593+0000", 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012580600000000001 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 14", 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.382745+0000", 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015200200000000001 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383376+0000", 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00063144299999999998 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383909+0000", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053270499999999996 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383947+0000", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7901000000000002e-05 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:11.383975+0000", 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.777e-05 2024-09-13T13:52:33.278 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:33.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:33.279 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:33.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:1992: done (7 left) 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:1993: done (6 left) 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2000: done (5 left) 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2001: write (excl) oid 38 current snap is 64 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2001: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2001: writing smithi15938319-38 from 588344 to 1219955 tid 1 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2001: writing smithi15938319-38 from 1689059 to 2430182 tid 2 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:2002: rollback oid 14 current snap is 64 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 14 to 50 2024-09-13T13:52:33.282 INFO:tasks.rados.rados.0.smithi159.stdout:1995: expect (ObjNum 360 snap 52 seq_num 360) 2024-09-13T13:52:33.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:33.283 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:33.280+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:33.300 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:33.299+0000 7f9b0c553640 -1 osd.12 168 handle_osd_map injecting map crc failure 2024-09-13T13:52:33.345 INFO:tasks.rados.rados.0.smithi159.stdout:1996: expect (ObjNum 220 snap 24 seq_num 220) 2024-09-13T13:52:33.381 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:33.380+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:33.401 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:33.401 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:33.401 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:33.407 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3533202 2024-09-13T13:52:33.410 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:33.483 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:33.482+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:33.493 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:33.556 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:33.556 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:33.556 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:33.556 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:33.556 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:33.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:33.580 INFO:tasks.rados.rados.0.smithi159.stdout:1997: expect (ObjNum 386 snap 60 seq_num 386) 2024-09-13T13:52:33.608 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:33.607+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:33.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:33.635 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:259 4.1 4:8e4bdd2a:::smithi15938319-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 140.768037838, 2024-09-13T13:52:33.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010862444000000001, 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 259 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:33.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898165+0000", 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898168+0000", 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2110000000000001e-06 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898179+0000", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0834e-05 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898197+0000", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8439000000000001e-05 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898206+0000", 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4199999999999996e-06 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898267+0000", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0430000000000002e-05 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898348+0000", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1366000000000007e-05 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908006+0000", 2024-09-13T13:52:33.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096577490000000002 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908606+0000", 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059992800000000001 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908925+0000", 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031962700000000001 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.908984+0000", 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8755999999999998e-05 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.909028+0000", 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3683999999999997e-05 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:266 4.1 4:8e40678c:::smithi15938319-17:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.674 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 140.752410223, 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012364559000000001, 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 266 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913793+0000", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913794+0000", 2024-09-13T13:52:33.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913799+0000", 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913810+0000", 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567999999999999e-05 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913818+0000", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0199999999999994e-06 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913878+0000", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0195999999999998e-05 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.913963+0000", 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4659000000000004e-05 2024-09-13T13:52:33.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925277+0000", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011314467 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.925692+0000", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041448599999999999 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926065+0000", 2024-09-13T13:52:33.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037329799999999999 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926115+0000", 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9867999999999998e-05 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.926158+0000", 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2874000000000003e-05 2024-09-13T13:52:33.680 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:278 4.1 4:81ab2613:::smithi15938319-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 140.743698548, 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014898105, 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.681 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 278 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922505+0000", 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922506+0000", 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922512+0000", 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999997e-06 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922528+0000", 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5710999999999999e-05 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922536+0000", 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4579999999999993e-06 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922594+0000", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7778999999999997e-05 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.922669+0000", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4753999999999996e-05 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.936619+0000", 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013950749 2024-09-13T13:52:33.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937127+0000", 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050706799999999995 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937301+0000", 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174055 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937361+0000", 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9973000000000003e-05 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.937403+0000", 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2330000000000003e-05 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:117 4.f e53/34 4:f883ee18:::smithi15938319-40:head v 53'56, mlcod=51'54)", 2024-09-13T13:52:33.687 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 124.91941239800001, 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011197554, 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.746791+0000", 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.11e-07 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749521+0000", 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027301410000000002 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749557+0000", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5321999999999998e-05 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749565+0000", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749633+0000", 2024-09-13T13:52:33.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7404999999999996e-05 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.749657+0000", 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4406e-05 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757925+0000", 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082682680000000005 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757988+0000", 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3021000000000005e-05 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1334 4.1 4:8b8235b2:::smithi15938319-16:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 123.704742119, 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22642320499999999, 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.692 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1334 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961461+0000", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961463+0000", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.858e-06 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961470+0000", 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1149999999999996e-06 2024-09-13T13:52:33.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961487+0000", 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6657e-05 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961513+0000", 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5834999999999999e-05 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961567+0000", 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.961571+0000", 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4479999999999996e-06 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966851+0000", 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052797390000000003 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:29.966910+0000", 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9574999999999998e-05 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:30.187884+0000", 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22097413199999999 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1494 4.1 4:8e4bdd2a:::smithi15938319-10:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e56)", 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 120.702940493, 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017102605999999999, 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1494 2024-09-13T13:52:33.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963263+0000", 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963265+0000", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.17e-06 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963273+0000", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.858e-06 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963291+0000", 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7982000000000002e-05 2024-09-13T13:52:33.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963307+0000", 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5936000000000001e-05 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963381+0000", 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4430000000000004e-05 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.963469+0000", 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7837000000000006e-05 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:32.980365+0000", 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016896392999999999 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.702 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1803 4.1 4:8bb695d2:::smithi15938319-48:c [sparse-read 0~2698372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.42401692599999, 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014432692, 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1803 2024-09-13T13:52:33.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242186+0000", 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242187+0000", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242190+0000", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8679999999999999e-06 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242196+0000", 2024-09-13T13:52:33.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2140000000000001e-06 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242201+0000", 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1699999999999996e-06 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242398+0000", 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197216 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242408+0000", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.234e-06 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250663+0000", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082552609999999998 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.250788+0000", 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012527299999999999 2024-09-13T13:52:33.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256619+0000", 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0058309190000000004 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1804 4.1 4:8bb695d2:::smithi15938319-48:c [read 0~3747928,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 112.423993567, 2024-09-13T13:52:33.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022225074000000001, 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1804 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:33.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242210+0000", 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5500000000000001e-06 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242220+0000", 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9739999999999996e-06 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242226+0000", 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2999999999999998e-06 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242436+0000", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021049799999999999 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242443+0000", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7499999999999997e-06 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256681+0000", 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014238195 2024-09-13T13:52:33.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.256754+0000", 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2206000000000006e-05 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.264435+0000", 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076811839999999998 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(osd.13.0:221 4.f e84/34 4:fb8e36a2:::smithi15938319-39:9 v 84'140, mlcod=84'134)", 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 92.839927367000001, 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19270847799999999, 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826276+0000", 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4000000000000002e-07 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.826280+0000", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4699999999999998e-06 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017940+0000", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19166003200000001 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.017956+0000", 2024-09-13T13:52:33.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5617e-05 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018053+0000", 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7136e-05 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018072+0000", 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9293999999999999e-05 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018951+0000", 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087949399999999998 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.018984+0000", 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2994999999999998e-05 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.718 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2815 4.1 4:8e40678c:::smithi15938319-17:head [write 699224~548358,stat] snapc 1b=[1b,1a,19,18,17,16,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.905057123000006, 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012616176, 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2815 2024-09-13T13:52:33.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761146+0000", 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.761147+0000", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.879999999999999e-07 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768547+0000", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073998249999999996 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768556+0000", 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0569999999999999e-06 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768563+0000", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7170000000000002e-06 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768619+0000", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6393e-05 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768710+0000", 2024-09-13T13:52:33.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0840000000000004e-05 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.768892+0000", 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000181502 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773631+0000", 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047397860000000002 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773661+0000", 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9853e-05 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773715+0000", 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3860999999999998e-05 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:10.773762+0000", 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7354000000000001e-05 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.724 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2854 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.809112807999995, 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015083855, 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.725 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2854 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857091+0000", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857095+0000", 2024-09-13T13:52:33.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5669999999999999e-06 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857099+0000", 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857103+0000", 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5429999999999998e-06 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857222+0000", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899400000000001 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857227+0000", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6400000000000002e-06 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.866914+0000", 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0096866360000000002 2024-09-13T13:52:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.867028+0000", 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011347499999999999 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872174+0000", 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005146851 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2855 4.1 4:8e40678c:::smithi15938319-17:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.809074946999999, 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016605034000000001, 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2855 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857128+0000", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857129+0000", 2024-09-13T13:52:33.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0200000000000002e-07 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857133+0000", 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.827e-06 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857141+0000", 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0309999999999998e-06 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857155+0000", 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3994e-05 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857287+0000", 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013179199999999999 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.857292+0000", 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7939999999999999e-06 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872200+0000", 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01490738 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.872266+0000", 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6420000000000004e-05 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:11.873733+0000", 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014672940000000001 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2907 4.1 4:8b8235b2:::smithi15938319-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 80.827971751999996, 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010086731, 2024-09-13T13:52:33.736 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2907 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838232+0000", 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8099999999999999e-07 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838236+0000", 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4059999999999998e-06 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838240+0000", 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7069999999999998e-06 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.838243+0000", 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9969999999999999e-06 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840154+0000", 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019110360000000001 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.840235+0000", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1001999999999995e-05 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848108+0000", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078728500000000007 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848183+0000", 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5477999999999998e-05 2024-09-13T13:52:33.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:12.848318+0000", 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134974 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3899 4.1 4:8bb695d2:::smithi15938319-48:head [write 561433~774904,stat] snapc 2f=[2f,2d,2c,2a,28,27,25,24,23,22,1e,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 36.371424312999999, 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011332106, 2024-09-13T13:52:33.742 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3899 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294779+0000", 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.294781+0000", 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1880000000000001e-06 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295899+0000", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011175060000000001 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295916+0000", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7275e-05 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295925+0000", 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2550000000000003e-06 2024-09-13T13:52:33.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295992+0000", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6973999999999999e-05 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.295999+0000", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299822+0000", 2024-09-13T13:52:33.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038227019999999999 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.299902+0000", 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0341999999999998e-05 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.300046+0000", 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014437299999999999 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.304078+0000", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040313459999999999 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.305972+0000", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018946200000000001 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306041+0000", 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8399999999999996e-05 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:57.306111+0000", 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0492000000000002e-05 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4169 4.1 4:81ab2613:::smithi15938319-29:head [read 0~3462209,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.579255637999999, 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012063419000000001, 2024-09-13T13:52:33.749 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4169 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086948+0000", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086949+0000", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086957+0000", 2024-09-13T13:52:33.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6350000000000006e-06 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086972+0000", 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5503e-05 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.086982+0000", 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0009e-05 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087052+0000", 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9832999999999994e-05 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087058+0000", 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7869999999999998e-06 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091635+0000", 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045774259999999999 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.091765+0000", 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012954199999999999 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099011+0000", 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072461840000000001 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4170 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.579197829999998, 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014864408000000001, 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4170 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087005+0000", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087006+0000", 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0100000000000005e-07 2024-09-13T13:52:33.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087008+0000", 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4200000000000001e-06 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087015+0000", 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087018+0000", 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7359999999999999e-06 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087126+0000", 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107847 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087134+0000", 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2849999999999995e-06 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099058+0000", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011923375 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.099212+0000", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015423100000000001 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101870+0000", 2024-09-13T13:52:33.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002657899 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4171 4.1 4:81ab2613:::smithi15938319-29:head [sparse-read 0~2430017] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.579162205999999, 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016120193000000001, 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.760 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4171 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087041+0000", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3700000000000001e-07 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087044+0000", 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0870000000000001e-06 2024-09-13T13:52:33.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087048+0000", 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3170000000000003e-06 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087053+0000", 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.609e-06 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087165+0000", 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111686 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.087169+0000", 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6600000000000001e-06 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.101916+0000", 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014746795 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.102043+0000", 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012732600000000001 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:06.103161+0000", 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011183759999999999 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.765 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.4571.0:4611 4.b e164/34 4:def5de2b:::smithi15938319-3:head v 164'366, mlcod=163'362)", 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.197514+0000", 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4686888890000001, 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012561256999999999, 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197514+0000", 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197514+0000", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.197515+0000", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8e-07 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.198013+0000", 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049874399999999997 2024-09-13T13:52:33.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.198048+0000", 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5135000000000002e-05 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.198056+0000", 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1609999999999997e-06 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.768 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.198112+0000", 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6638000000000002e-05 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.198135+0000", 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2957999999999999e-05 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210050+0000", 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011914275 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210076+0000", 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6166e-05 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4616 4.3 4:c1ae59bf:::smithi15938319-45:33 [sparse-read 0~2167537] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.199891+0000", 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4663121759999997, 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010188515, 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.770 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4616 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199891+0000", 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199891+0000", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199891+0000", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3500000000000002e-07 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199893+0000", 2024-09-13T13:52:33.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.846e-06 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199897+0000", 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1470000000000003e-06 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199901+0000", 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3009999999999998e-06 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.205526+0000", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0056253659999999997 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.205623+0000", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7239999999999997e-05 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210080+0000", 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00445628 2024-09-13T13:52:33.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:4617 4.3 4:c1ae59bf:::smithi15938319-45:33 [read 494014~2464051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:52:28.199920+0000", 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.4662835279999999, 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010946948999999999, 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:33.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4617 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199920+0000", 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199920+0000", 2024-09-13T13:52:33.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199920+0000", 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.91e-07 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199922+0000", 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0260000000000002e-06 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:33.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199926+0000", 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2799999999999997e-06 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.199929+0000", 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9639999999999999e-06 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210069+0000", 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010139568 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210134+0000", 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4826000000000006e-05 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:52:28.210867+0000", 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00073299400000000005 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:33.779 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:33.782 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:33.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:33.783 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:33.752+0000 7fe0fbe07640 -1 osd.2 168 handle_osd_map injecting map crc failure 2024-09-13T13:52:33.783 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:33.784 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:33.780+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:33.784 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:33.785 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:33.785 INFO:tasks.rados.rados.0.smithi159.stdout:2001: finishing write tid 1 to smithi15938319-38 2024-09-13T13:52:33.785 INFO:tasks.rados.rados.0.smithi159.stdout:2001: finishing write tid 2 to smithi15938319-38 2024-09-13T13:52:33.785 INFO:tasks.rados.rados.0.smithi159.stdout:2001: finishing write tid 3 to smithi15938319-38 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2001: finishing write tid 4 to smithi15938319-38 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2001: oid 38 updating version 0 to 313 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2001: oid 38 updating version 313 to 314 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2001: oid 38 version 314 is already newer than 312 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 38 v 314 (ObjNum 392 snap 64 seq_num 392) dirty exists 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2001: left oid 38 (ObjNum 392 snap 64 seq_num 392) 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2002: finishing rollback tid 0 to smithi15938319-14 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2002: finishing rollback tid 1 to smithi15938319-14 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:2002: finishing rollback tid 2 to smithi15938319-14 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 270 (ObjNum 315 snap 45 seq_num 315) dirty exists 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:1995: done (6 left) 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:1996: done (5 left) 2024-09-13T13:52:33.786 INFO:tasks.rados.rados.0.smithi159.stdout:1997: done (4 left) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:1998: done (3 left) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:1999: done (2 left) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2001: done (1 left) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2002: done (0 left) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2003: tier_promote oid 48 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2004: read oid 26 snap 50 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2004: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2005: write oid 16 current snap is 64 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2005: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2005: writing smithi15938319-16 from 605151 to 1312011 tid 1 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2005: writing smithi15938319-16 from 2031188 to 2605151 tid 2 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2006: tier_promote oid 21 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2007: read oid 10 snap -1 2024-09-13T13:52:33.787 INFO:tasks.rados.rados.0.smithi159.stdout:2007: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2008: read oid 29 snap -1 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2008: expect (ObjNum 366 snap 54 seq_num 366) 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2009: read oid 50 snap -1 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2009: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2010: tier_promote oid 5 2024-09-13T13:52:33.788 INFO:tasks.rados.rados.0.smithi159.stdout:2004: expect (ObjNum 311 snap 45 seq_num 311) 2024-09-13T13:52:33.789 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2861092 2024-09-13T13:52:33.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:33.793 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:33.814 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:33.814 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:33.814 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1926241 2024-09-13T13:52:33.834 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:33.856 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:33.876 INFO:tasks.rados.rados.0.smithi159.stdout:2009: expect (ObjNum 349 snap 49 seq_num 349) 2024-09-13T13:52:33.879 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:33.879 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:33.881 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:33.881+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:33.888 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1708811 2024-09-13T13:52:33.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:33.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:33.920 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:33.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:33.926 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:33.927 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:33.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:33.941 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:33.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:33.951 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:33.950+0000 7fe857f4f640 -1 osd.7 169 handle_osd_map injecting map crc failure 2024-09-13T13:52:33.951 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:33.951+0000 7f586a715640 -1 osd.9 168 handle_osd_map injecting map crc failure 2024-09-13T13:52:33.963 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:33.963 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:33.982 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:33.981+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:33.984 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:33.984 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:33.996 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1857221 2024-09-13T13:52:34.004 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:34.005 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:34.025 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:34.026 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:34.028 INFO:tasks.rados.rados.0.smithi159.stdout:2003: got (0) Success 2024-09-13T13:52:34.028 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 48 v 463 (ObjNum 289 snap 0 seq_num 0) clean dne 2024-09-13T13:52:34.028 INFO:tasks.rados.rados.0.smithi159.stdout:2005: finishing write tid 1 to smithi15938319-16 2024-09-13T13:52:34.028 INFO:tasks.rados.rados.0.smithi159.stdout:2005: finishing write tid 2 to smithi15938319-16 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2003: done (7 left) 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2004: done (6 left) 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2009: done (5 left) 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2011: tier_promote oid 1 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2010: got (0) Success 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 5 v 247 (ObjNum 336 snap 47 seq_num 336) dirty exists 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: finishing write tid 3 to smithi15938319-16 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: finishing write tid 4 to smithi15938319-16 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: oid 16 updating version 0 to 467 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: oid 16 version 467 is already newer than 466 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: oid 16 updating version 467 to 468 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 468 (ObjNum 393 snap 64 seq_num 393) dirty exists 2024-09-13T13:52:34.029 INFO:tasks.rados.rados.0.smithi159.stdout:2005: left oid 16 (ObjNum 393 snap 64 seq_num 393) 2024-09-13T13:52:34.030 INFO:tasks.rados.rados.0.smithi159.stdout:2007: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:34.030 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:34.028+0000 7f4ffdd7e640 -1 osd.1 168 handle_osd_map injecting map crc failure 2024-09-13T13:52:34.046 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:34.046 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:34.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:34.051+0000 7f44c37fe640 0 -- 172.21.15.27:0/2671392113 >> v2:172.21.15.159:6805/4073091144 conn(0x7f44c4222480 msgr2=0x7f44c4224870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T13:52:34.072 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:34.076 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:34.077 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1129260 2024-09-13T13:52:34.078 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.078 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.084 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:34.082+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:34.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:34.108 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:34.154 INFO:tasks.rados.rados.0.smithi159.stdout:2006: got (0) Success 2024-09-13T13:52:34.154 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 21 v 147 (ObjNum 305 snap 43 seq_num 305) dirty exists 2024-09-13T13:52:34.154 INFO:tasks.rados.rados.0.smithi159.stdout:2008: expect (ObjNum 366 snap 54 seq_num 366) 2024-09-13T13:52:34.154 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:34.184 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:34.184+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:34.189 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.189 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:34.201 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 564025 2024-09-13T13:52:34.246 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1289245 2024-09-13T13:52:34.270 INFO:tasks.rados.rados.0.smithi159.stdout:2011: got (0) Success 2024-09-13T13:52:34.270 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 1 v 224 (ObjNum 338 snap 47 seq_num 338) dirty exists 2024-09-13T13:52:34.270 INFO:tasks.rados.rados.0.smithi159.stdout:2005: done (5 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2006: done (4 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2007: done (3 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2008: done (2 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2010: done (1 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2011: done (0 left) 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2012: read oid 22 snap -1 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2012: expect deleted 2024-09-13T13:52:34.271 INFO:tasks.rados.rados.0.smithi159.stdout:2013: set_chunk oid 17 target oid 2024-09-13T13:52:34.274 INFO:tasks.rados.rados.0.smithi159.stdout:2013: set_chunk oid 17 offset: 3383808 length: 232960 target oid 5d232a1b89304e7a3a5edb69a0349b8b45976e7bf44269593d0f57ad5049be6c offset: 0 snap 53 2024-09-13T13:52:34.274 INFO:tasks.rados.rados.0.smithi159.stdout:2014: write oid 42 current snap is 64 2024-09-13T13:52:34.275 INFO:tasks.rados.rados.0.smithi159.stdout:2014: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-13T13:52:34.281 INFO:tasks.rados.rados.0.smithi159.stdout:2014: writing smithi15938319-42 from 621958 to 1404067 tid 1 2024-09-13T13:52:34.286 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:34.284+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:34.288 INFO:tasks.rados.rados.0.smithi159.stdout:2014: writing smithi15938319-42 from 2089670 to 2621958 tid 2 2024-09-13T13:52:34.288 INFO:tasks.rados.rados.0.smithi159.stdout:2015: read oid 48 snap -1 2024-09-13T13:52:34.288 INFO:tasks.rados.rados.0.smithi159.stdout:2015: expect deleted 2024-09-13T13:52:34.288 INFO:tasks.rados.rados.0.smithi159.stdout:2016: set_chunk oid 9 target oid 2024-09-13T13:52:34.291 INFO:tasks.rados.rados.0.smithi159.stdout:2016: set_chunk oid 9 offset: 216064 length: 70144 target oid 160fe14a04c81fc94e208850df8283f9a3736840a418ca51babbd6f1056ffbcf offset: 0 snap -1 2024-09-13T13:52:34.291 INFO:tasks.rados.rados.0.smithi159.stdout:2017: set_chunk oid 16 target oid 2024-09-13T13:52:34.294 INFO:tasks.rados.rados.0.smithi159.stdout:2017: set_chunk oid 16 offset: 414720 length: 101376 target oid 8e19a1f53221ff05354b2f464fec7c757f816442c6294db19c1cb2e08f707a5e offset: 0 snap -1 2024-09-13T13:52:34.294 INFO:tasks.rados.rados.0.smithi159.stdout:2018: set_chunk oid 46 target oid 2024-09-13T13:52:34.306 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.306 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:34.318 INFO:tasks.rados.rados.0.smithi159.stdout:2018: set_chunk oid 46 offset: 807936 length: 2518528 target oid ab3ec006ff3a9b32de788f57468966c6bd3ee5043af0361885bbbdb3aa38cd4e offset: 0 snap 50 2024-09-13T13:52:34.318 INFO:tasks.rados.rados.0.smithi159.stdout:2019: read oid 18 snap -1 2024-09-13T13:52:34.318 INFO:tasks.rados.rados.0.smithi159.stdout:2019: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:52:34.318 INFO:tasks.rados.rados.0.smithi159.stdout:2020: write (excl) oid 7 current snap is 64 2024-09-13T13:52:34.318 INFO:tasks.rados.rados.0.smithi159.stdout:2020: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-13T13:52:34.323 INFO:tasks.rados.rados.0.smithi159.stdout:2020: writing smithi15938319-7 from 638765 to 1096123 tid 1 2024-09-13T13:52:34.330 INFO:tasks.rados.rados.0.smithi159.stdout:2020: writing smithi15938319-7 from 1748152 to 2352955 tid 2 2024-09-13T13:52:34.330 INFO:tasks.rados.rados.0.smithi159.stdout:2021: write oid 40 current snap is 64 2024-09-13T13:52:34.331 INFO:tasks.rados.rados.0.smithi159.stdout:2021: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-13T13:52:34.337 INFO:tasks.rados.rados.0.smithi159.stdout:2021: writing smithi15938319-40 from 655572 to 1304532 tid 1 2024-09-13T13:52:34.345 INFO:tasks.rados.rados.0.smithi159.stdout:2021: writing smithi15938319-40 from 1922987 to 2655572 tid 2 2024-09-13T13:52:34.345 INFO:tasks.rados.rados.0.smithi159.stdout:2022: rollback oid 8 current snap is 64 2024-09-13T13:52:34.345 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 8 to 49 2024-09-13T13:52:34.345 INFO:tasks.rados.rados.0.smithi159.stdout:2013: finishing set_chunk to oid 17 2024-09-13T13:52:34.345 INFO:tasks.rados.rados.0.smithi159.stdout:2014: finishing write tid 1 to smithi15938319-42 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: finishing write tid 2 to smithi15938319-42 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: finishing write tid 3 to smithi15938319-42 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: finishing write tid 4 to smithi15938319-42 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: oid 42 updating version 0 to 249 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: oid 42 updating version 249 to 251 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: oid 42 version 251 is already newer than 250 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 251 (ObjNum 394 snap 64 seq_num 394) dirty exists 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2014: left oid 42 (ObjNum 394 snap 64 seq_num 394) 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2016: finishing set_chunk to oid 9 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 9 v 249 (ObjNum 157 snap 13 seq_num 157) dirty exists 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2017: finishing set_chunk to oid 16 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 16 v 468 (ObjNum 393 snap 64 seq_num 393) dirty exists 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2018: finishing set_chunk to oid 46 2024-09-13T13:52:34.346 INFO:tasks.rados.rados.0.smithi159.stdout:2012: done (10 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2013: done (9 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2014: done (8 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2015: done (7 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2016: done (6 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2017: done (5 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2018: done (4 left) 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2023: read oid 43 snap 60 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2023: expect deleted 2024-09-13T13:52:34.347 INFO:tasks.rados.rados.0.smithi159.stdout:2024: set_chunk oid 31 target oid 2024-09-13T13:52:34.365 INFO:tasks.rados.rados.0.smithi159.stdout:2024: set_chunk oid 31 offset: 371200 length: 1722880 target oid 1822e8b8343c5fe41a83ccce55da3ebcca91d7615acc7092fa52da4eb88df5a2 offset: 0 snap 52 2024-09-13T13:52:34.365 INFO:tasks.rados.rados.0.smithi159.stdout:2025: tier_promote oid 45 2024-09-13T13:52:34.365 INFO:tasks.rados.rados.0.smithi159.stdout:2019: expect (ObjNum 266 snap 32 seq_num 266) 2024-09-13T13:52:34.386 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:34.386+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:34.424 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.424 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:34.483 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 437552 2024-09-13T13:52:34.487 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:34.486+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:34.536 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.536 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.542 INFO:tasks.rados.rados.0.smithi159.stdout:2020: finishing write tid 1 to smithi15938319-7 2024-09-13T13:52:34.542 INFO:tasks.rados.rados.0.smithi159.stdout:2020: finishing write tid 2 to smithi15938319-7 2024-09-13T13:52:34.542 INFO:tasks.rados.rados.0.smithi159.stdout:2020: finishing write tid 3 to smithi15938319-7 2024-09-13T13:52:34.542 INFO:tasks.rados.rados.0.smithi159.stdout:2020: finishing write tid 4 to smithi15938319-7 2024-09-13T13:52:34.542 INFO:tasks.rados.rados.0.smithi159.stdout:2020: oid 7 updating version 0 to 376 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2020: oid 7 updating version 376 to 378 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2020: oid 7 version 378 is already newer than 377 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 378 (ObjNum 395 snap 64 seq_num 395) dirty exists 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2020: left oid 7 (ObjNum 395 snap 64 seq_num 395) 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2021: finishing write tid 1 to smithi15938319-40 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2019: done (6 left) 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2020: done (5 left) 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2026: tier_evict oid 16 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2026: tier_evict oid 16 snap -1 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2022: finishing rollback tid 0 to smithi15938319-8 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2022: finishing rollback tid 1 to smithi15938319-8 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:2022: finishing rollback tid 2 to smithi15938319-8 2024-09-13T13:52:34.543 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 8 v 255 (ObjNum 317 snap 46 seq_num 317) dirty exists 2024-09-13T13:52:34.544 INFO:tasks.rados.rados.0.smithi159.stdout:2021: finishing write tid 2 to smithi15938319-40 2024-09-13T13:52:34.544 INFO:tasks.rados.rados.0.smithi159.stdout:2021: finishing write tid 3 to smithi15938319-40 2024-09-13T13:52:34.544 INFO:tasks.rados.rados.0.smithi159.stdout:2022: done (5 left) 2024-09-13T13:52:34.544 INFO:tasks.rados.rados.0.smithi159.stdout:2023: done (4 left) 2024-09-13T13:52:34.544 INFO:tasks.rados.rados.0.smithi159.stdout:2027: set_chunk oid 3 target oid 2024-09-13T13:52:34.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:34.551 INFO:tasks.rados.rados.0.smithi159.stdout:2027: set_chunk oid 3 offset: 741888 length: 1379328 target oid 680998d0f4c852671e57fbc3d5c244bcdefa02b441f4f70c4d2c44d80b37defe offset: 0 snap -1 2024-09-13T13:52:34.551 INFO:tasks.rados.rados.0.smithi159.stdout:2028: snap_create 2024-09-13T13:52:34.551 INFO:tasks.rados.rados.0.smithi159.stdout:2021: finishing write tid 4 to smithi15938319-40 2024-09-13T13:52:34.551 INFO:tasks.rados.rados.0.smithi159.stdout:2021: oid 40 updating version 0 to 295 2024-09-13T13:52:34.551 INFO:tasks.rados.rados.0.smithi159.stdout:2021: oid 40 updating version 295 to 296 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:2021: oid 40 updating version 296 to 297 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 297 (ObjNum 396 snap 64 seq_num 396) dirty exists 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:2021: left oid 40 (ObjNum 396 snap 64 seq_num 396) 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:2024: finishing set_chunk to oid 31 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 371200 length: 1722880 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 31 v 0 (ObjNum 138 snap 11 seq_num 138) dirty exists 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:2025: got (0) Success 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 45 v 66 (ObjNum 129 snap 0 seq_num 1540572888) clean dne 2024-09-13T13:52:34.552 INFO:tasks.rados.rados.0.smithi159.stdout:2026: got (22) Invalid argument 2024-09-13T13:52:34.553 INFO:tasks.rados.rados.0.smithi159.stdout:2027: finishing set_chunk to oid 3 2024-09-13T13:52:34.553 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 3 v 368 (ObjNum 384 snap 60 seq_num 384) dirty exists 2024-09-13T13:52:34.590 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:34.589+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:34.647 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.647 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.655 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:34.689 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:34.688+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:34.790 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:34.790+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:34.799 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.799 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:34.891 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:34.890+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:34.909 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:34.910 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:34.918 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:34.991 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:34.991+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:35.023 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.023 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.032 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:35.092 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:35.091+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:35.176 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.177 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.185 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:35.193 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:35.192+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:35.210 INFO:tasks.rados.rados.0.smithi159.stdout:2021: done (5 left) 2024-09-13T13:52:35.210 INFO:tasks.rados.rados.0.smithi159.stdout:2024: done (4 left) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2025: done (3 left) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2026: done (2 left) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2027: done (1 left) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2028: done (0 left) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2029: read oid 12 snap -1 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2029: expect (ObjNum 388 snap 60 seq_num 388) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2030: read oid 3 snap -1 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2030: expect (ObjNum 384 snap 60 seq_num 384) 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2031: rollback oid 42 current snap is 65 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:rollback oid 42 to 63 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2032: read oid 47 snap 54 2024-09-13T13:52:35.211 INFO:tasks.rados.rados.0.smithi159.stdout:2032: expect (ObjNum 370 snap 54 seq_num 370) 2024-09-13T13:52:35.212 INFO:tasks.rados.rados.0.smithi159.stdout:2033: read oid 13 snap -1 2024-09-13T13:52:35.212 INFO:tasks.rados.rados.0.smithi159.stdout:2033: expect (ObjNum 372 snap 54 seq_num 372) 2024-09-13T13:52:35.212 INFO:tasks.rados.rados.0.smithi159.stdout:2034: read oid 17 snap -1 2024-09-13T13:52:35.212 INFO:tasks.rados.rados.0.smithi159.stdout:2034: expect (ObjNum 369 snap 54 seq_num 369) 2024-09-13T13:52:35.212 INFO:tasks.rados.rados.0.smithi159.stdout:2035: read oid 46 snap -1 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2035: expect (ObjNum 345 snap 47 seq_num 345) 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2036: read oid 24 snap -1 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2036: expect (ObjNum 339 snap 47 seq_num 339) 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2037: read oid 10 snap 48 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2037: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2038: read oid 9 snap -1 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2038: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2039: tier_promote oid 7 2024-09-13T13:52:35.213 INFO:tasks.rados.rados.0.smithi159.stdout:2040: set_chunk oid 14 target oid 2024-09-13T13:52:35.214 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:35.209+0000 7f9b0e557640 -1 osd.12 170 handle_osd_map injecting map crc failure 2024-09-13T13:52:35.215 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:35.214+0000 7f4ffbd7a640 -1 osd.1 170 handle_osd_map injecting map crc failure 2024-09-13T13:52:35.248 INFO:tasks.rados.rados.0.smithi159.stdout:2040: set_chunk oid 14 offset: 443392 length: 2330112 target oid b0113e7a02350cd609494306f7d6df1062f8a1b4e67f5fb9ee4f693b5627e152 offset: 0 snap -1 2024-09-13T13:52:35.248 INFO:tasks.rados.rados.0.smithi159.stdout:2041: write oid 36 current snap is 65 2024-09-13T13:52:35.248 INFO:tasks.rados.rados.0.smithi159.stdout:2041: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-13T13:52:35.255 INFO:tasks.rados.rados.0.smithi159.stdout:2041: writing smithi15938319-36 from 672379 to 1396588 tid 1 2024-09-13T13:52:35.261 INFO:tasks.rados.rados.0.smithi159.stdout:2041: writing smithi15938319-36 from 1865116 to 2473588 tid 2 2024-09-13T13:52:35.262 INFO:tasks.rados.rados.0.smithi159.stdout:2042: write oid 37 current snap is 65 2024-09-13T13:52:35.262 INFO:tasks.rados.rados.0.smithi159.stdout:2042: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-13T13:52:35.269 INFO:tasks.rados.rados.0.smithi159.stdout:2042: writing smithi15938319-37 from 689186 to 1488644 tid 1 2024-09-13T13:52:35.276 INFO:tasks.rados.rados.0.smithi159.stdout:2042: writing smithi15938319-37 from 1923598 to 2675728 tid 2 2024-09-13T13:52:35.276 INFO:tasks.rados.rados.0.smithi159.stdout:2043: snap_remove snap 32 2024-09-13T13:52:35.294 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:35.294+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:35.297 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.297 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.306 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:35.395 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:35.394+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:35.449 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.449 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.458 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:35.497 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:35.496+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:35.561 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.561 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.571 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:35.598 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:35.597+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:35.673 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.673 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.682 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:35.698 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:35.697+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:35.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:52:35.785 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.785 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.793 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:35.830 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:35.830+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:35.896 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:35.897 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:35.900 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:35.899+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:35.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:36.001 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:36.000+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:36.007 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:36.007 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:36.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:36.077 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:36.101 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:36.100+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:36.118 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:36.118 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:36.126 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:36.202 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:36.201+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:36.207 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:52:36.208 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:36.208+0000 7faeac214640 -1 osd.8 171 handle_osd_map injecting map crc failure 2024-09-13T13:52:36.208 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:36.207+0000 7f26bcfe2640 -1 osd.3 171 handle_osd_map injecting map crc failure 2024-09-13T13:52:36.211 INFO:tasks.rados.rados.0.smithi159.stdout:2043: done (14 left) 2024-09-13T13:52:36.211 INFO:tasks.rados.rados.0.smithi159.stdout:2044: tier_evict oid 20 2024-09-13T13:52:36.211 INFO:tasks.rados.rados.0.smithi159.stdout:2044: tier_evict oid 20 snap -1 2024-09-13T13:52:36.211 INFO:tasks.rados.rados.0.smithi159.stdout:2033: expect (ObjNum 372 snap 54 seq_num 372) 2024-09-13T13:52:36.211 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:36.214 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:36.215 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:36.235 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:36.235 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:36.238 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:36.245 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:36.256 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:36.259 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:36.259 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:36.280 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:36.280 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:36.280 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:36.282 INFO:tasks.rados.rados.0.smithi159.stdout:2035: expect (ObjNum 345 snap 47 seq_num 345) 2024-09-13T13:52:36.302 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:36.302 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:36.303 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:36.302+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:36.310 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3044403 2024-09-13T13:52:36.323 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:36.324 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:36.327 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:36.346 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:36.347 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:36.360 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:36.360 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:36.367 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:36.367 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:36.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:36.391 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:36.398 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 990017 2024-09-13T13:52:36.403 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:36.402+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:36.413 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:36.413 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:36.434 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:36.434 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:36.444 INFO:tasks.rados.rados.0.smithi159.stdout:2029: expect (ObjNum 388 snap 60 seq_num 388) 2024-09-13T13:52:36.455 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:36.455 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:36.476 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:36.477 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:36.480 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:36.480 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:36.480 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:36.481 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:36.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:36.498 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:36.502 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1724818 2024-09-13T13:52:36.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:36.502+0000 7fd1b1d83640 0 -- 172.21.15.27:0/156153625 >> v2:172.21.15.159:6805/4073091144 conn(0x7fd1ac220630 msgr2=0x7fd1ac222a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:36.503 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:36.503+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:36.516 INFO:tasks.rados.rados.0.smithi159.stdout:2034: expect (ObjNum 369 snap 54 seq_num 369) 2024-09-13T13:52:36.524 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:36.524 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:36.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:36.605 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:36.604+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:36.605 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:36.605 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:36.605 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:36.605 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:36.605 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:36.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:36.621 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 1899721 2024-09-13T13:52:36.674 INFO:tasks.rados.rados.0.smithi159.stdout:2031: finishing rollback tid 0 to smithi15938319-42 2024-09-13T13:52:36.674 INFO:tasks.rados.rados.0.smithi159.stdout:2031: finishing rollback tid 1 to smithi15938319-42 2024-09-13T13:52:36.674 INFO:tasks.rados.rados.0.smithi159.stdout:2031: finishing rollback tid 2 to smithi15938319-42 2024-09-13T13:52:36.674 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 42 v 259 (ObjNum 380 snap 59 seq_num 380) dirty exists 2024-09-13T13:52:36.674 INFO:tasks.rados.rados.0.smithi159.stdout:2032: expect (ObjNum 370 snap 54 seq_num 370) 2024-09-13T13:52:36.691 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 868704 2024-09-13T13:52:36.706 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:36.705+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:36.717 INFO:tasks.rados.rados.0.smithi159.stdout:2036: expect (ObjNum 339 snap 47 seq_num 339) 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.722 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:52:36.723 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:36.723 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 132.36640851499999, 2024-09-13T13:52:36.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:52:36.723 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.723 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:36.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:52:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.729 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:52:36.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:52:36.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:52:36.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 131.72695633399999, 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.733 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:52:36.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.737 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 131.72435937700001, 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:36.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.739 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:52:36.740 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 127.98464963399999, 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:52:36.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:36.745 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 127.97120490899999, 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:52:36.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 115.076258196, 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:52:36.749 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:52:36.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.753 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 115.032543112, 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:52:36.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 114.83872952900001, 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:52:36.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.761 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 104.95833768599999, 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.762 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:52:36.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:52:36.766 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.743246341, 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:52:36.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.771 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.74236791200001, 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:52:36.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:52:36.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:52:36.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 101.718652414, 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.776 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:52:36.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:52:36.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.945290365000005, 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:52:36.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:52:36.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.944689475999994, 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.785 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.786 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.787 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:52:36.788 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.789 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.937536898999994, 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:36.790 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:52:36.791 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.792 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.631772484999999, 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.796 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.797 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.798 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 99.624140952999994, 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.799 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:52:36.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.801 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.802 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.912669809999997, 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.803 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.804 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:52:36.805 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:52:36.806 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.911734843000005, 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.808 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:52:36.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:52:36.810 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:52:36.811 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:36.812 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.901052008999997, 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:36.813 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.814 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:36.815 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.816 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.817 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:36.817 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:36.817 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:36.818 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:36.818 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:36.817+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:36.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:36.919 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:36.918+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:36.923 INFO:tasks.rados.rados.0.smithi159.stdout:2030: expect (ObjNum 384 snap 60 seq_num 384) 2024-09-13T13:52:36.923 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:36.925 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:36.925 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:36.925 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:36.925 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:36.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:37.019 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:37.018+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:37.040 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:37.040 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:37.041 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:37.041 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:37.041 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:37.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:37.079 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 94713 2024-09-13T13:52:37.120 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:37.119+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:37.156 INFO:tasks.rados.rados.0.smithi159.stdout:2039: got (0) Success 2024-09-13T13:52:37.156 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 7 v 378 (ObjNum 395 snap 64 seq_num 395) dirty exists 2024-09-13T13:52:37.157 INFO:tasks.rados.rados.0.smithi159.stdout:2037: expect (ObjNum 302 snap 43 seq_num 302) 2024-09-13T13:52:37.157 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:19 4.a 4:5239d073:::smithi15938319-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.99076959199999, 2024-09-13T13:52:37.158 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10114572099999999, 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 19 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.159 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165611+0000", 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1199999999999998e-07 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.160 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165613+0000", 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9690000000000001e-06 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165618+0000", 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6600000000000003e-06 2024-09-13T13:52:37.161 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165621+0000", 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2890000000000002e-06 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167156+0000", 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001534854 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.162 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167202+0000", 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5479000000000003e-05 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167270+0000", 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8245999999999997e-05 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.169432+0000", 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002162204 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.163 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266661+0000", 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.097229382000000003 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266703+0000", 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1498999999999997e-05 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.164 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266757+0000", 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3727000000000002e-05 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:20 4.a 4:5239d073:::smithi15938319-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.99074359900001, 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.10137834499999999, 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 20 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.166 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165637+0000", 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1600000000000002e-07 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165639+0000", 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.564e-06 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165644+0000", 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3489999999999999e-06 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.165650+0000", 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6030000000000004e-06 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167441+0000", 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001791083 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.167526+0000", 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5638999999999998e-05 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.170 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.099281895999999994 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.266896+0000", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7897000000000002e-05 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.267015+0000", 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000118998 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.171 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:55 4.2 e41/34 4:4dc7b2b1:::smithi15938319-14:head v 41'6, mlcod=41'2)", 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.885434777, 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16792323200000001, 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.172 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270946+0000", 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5199999999999996e-07 2024-09-13T13:52:37.173 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.270955+0000", 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6920000000000008e-06 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438354+0000", 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.16739875200000001 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.174 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438360+0000", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5540000000000003e-06 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438417+0000", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7111999999999998e-05 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438434+0000", 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6580999999999999e-05 2024-09-13T13:52:37.175 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438842+0000", 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000408231 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.438869+0000", 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6658000000000001e-05 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.176 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:151 4.a 4:5c84836d:::smithi15938319-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.58939960000001, 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12162272, 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.177 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 151 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.178 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566981+0000", 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7399999999999998e-07 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566983+0000", 2024-09-13T13:52:37.179 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9850000000000001e-06 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566987+0000", 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1810000000000003e-06 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566990+0000", 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9689999999999999e-06 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.180 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567719+0000", 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00072900899999999997 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567790+0000", 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0678000000000003e-05 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.181 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567876+0000", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5432000000000005e-05 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568355+0000", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00047971800000000002 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688526+0000", 2024-09-13T13:52:37.182 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.12017045799999999 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688561+0000", 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5661000000000003e-05 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688604+0000", 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2154999999999999e-05 2024-09-13T13:52:37.183 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.184 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:152 4.a 4:5c84836d:::smithi15938319-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.58938509999999, 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.121754195, 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 152 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566995+0000", 2024-09-13T13:52:37.186 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566996+0000", 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8200000000000001e-07 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.566997+0000", 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.187 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567000+0000", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1049999999999999e-06 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.567003+0000", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.695e-06 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568025+0000", 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010225810000000001 2024-09-13T13:52:37.188 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.568069+0000", 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3998999999999997e-05 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688648+0000", 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.120578413 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.189 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688691+0000", 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3560000000000003e-05 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688750+0000", 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8242999999999998e-05 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.190 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:151 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.58713166000001, 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119365419, 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.191 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569249+0000", 2024-09-13T13:52:37.192 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.89e-07 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.569252+0000", 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4059999999999999e-06 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.193 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688458+0000", 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.119206837 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688463+0000", 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5090000000000004e-06 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688516+0000", 2024-09-13T13:52:37.194 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.3143000000000003e-05 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688522+0000", 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2280000000000003e-06 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.688614+0000", 2024-09-13T13:52:37.195 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1906999999999999e-05 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:187 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.46470837199999, 2024-09-13T13:52:37.196 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20597288799999999, 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 187 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691672+0000", 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.32e-07 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.198 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691675+0000", 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1550000000000001e-06 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691680+0000", 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8690000000000003e-06 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691683+0000", 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.106e-06 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.199 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692451+0000", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076873800000000004 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692513+0000", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1836999999999998e-05 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692609+0000", 2024-09-13T13:52:37.200 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5468000000000001e-05 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693303+0000", 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00069385300000000003 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.201 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897565+0000", 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20426237899999999 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897604+0000", 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9360000000000003e-05 2024-09-13T13:52:37.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.203 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897645+0000", 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0691000000000001e-05 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.204 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:188 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.464677553, 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20609751900000001, 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.205 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 188 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.206 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691703+0000", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691705+0000", 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.891e-06 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691709+0000", 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9600000000000002e-06 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.691712+0000", 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0529999999999998e-06 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692750+0000", 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010381120000000001 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.692792+0000", 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1696000000000002e-05 2024-09-13T13:52:37.209 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897686+0000", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20489396500000001 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897739+0000", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2904e-05 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897800+0000", 2024-09-13T13:52:37.210 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1631000000000007e-05 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:187 4.a e41/34 ondisk, result = 0)", 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.462519445, 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.20379767600000001, 2024-09-13T13:52:37.211 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:37.212 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693861+0000", 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6e-07 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.693863+0000", 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.736e-06 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.213 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897456+0000", 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.203593464 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897466+0000", 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.3139999999999993e-06 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897553+0000", 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7312999999999996e-05 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.214 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897561+0000", 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8199999999999997e-06 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.897659+0000", 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7769000000000005e-05 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.215 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(osd.10.22:11 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.252985247, 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.85271136599999997, 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.216 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903395+0000", 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.217 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903397+0000", 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.903408+0000", 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1853e-05 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.218 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.752993+0000", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.84958473099999998 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753012+0000", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8966000000000001e-05 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753480+0000", 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00046750600000000002 2024-09-13T13:52:37.219 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.753597+0000", 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000117726 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.756107+0000", 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025092510000000001 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.220 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:296 4.a 4:5e6df132:::smithi15938319-47:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 143.38700771500001, 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.30427271, 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 296 2024-09-13T13:52:37.221 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769373+0000", 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769374+0000", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6810000000000001e-06 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769384+0000", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4059999999999994e-06 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769398+0000", 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4351e-05 2024-09-13T13:52:37.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769417+0000", 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9211999999999998e-05 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769495+0000", 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7540999999999995e-05 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.224 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.769608+0000", 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000113238 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.778865+0000", 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0092566579999999992 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.779424+0000", 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00055964800000000005 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.225 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073537+0000", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29411226099999999 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073598+0000", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1576999999999997e-05 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073645+0000", 2024-09-13T13:52:37.226 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7136999999999999e-05 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:296 4.a e42/34 ondisk, result = 0)", 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 143.37333200099999, 2024-09-13T13:52:37.227 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29060506800000002, 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783048+0000", 2024-09-13T13:52:37.228 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783049+0000", 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6199999999999997e-07 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:13.783057+0000", 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.3849999999999996e-06 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.229 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073438+0000", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.29038136399999998 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073447+0000", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.9840000000000007e-06 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073519+0000", 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1914000000000003e-05 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.230 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073530+0000", 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0856000000000001e-05 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:14.073653+0000", 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012380299999999999 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.231 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2548 4.a 4:5e6df132:::smithi15938319-47:head [write 645435~451223] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.357430417000003, 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0449398489999999, 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2548 2024-09-13T13:52:37.232 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798950+0000", 2024-09-13T13:52:37.233 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.798956+0000", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0630000000000002e-06 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799524+0000", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00056807299999999999 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799551+0000", 2024-09-13T13:52:37.234 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6452999999999999e-05 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799565+0000", 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4141000000000001e-05 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799643+0000", 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8492000000000007e-05 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:37.235 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.799649+0000", 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5840000000000002e-06 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805208+0000", 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0055590989999999996 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.805222+0000", 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.415e-05 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.236 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822008+0000", 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0167862679999999 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823722+0000", 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001713963 2024-09-13T13:52:37.237 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838587+0000", 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014864936 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838649+0000", 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1982000000000004e-05 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.238 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.838815+0000", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016573900000000001 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843774+0000", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0049594920000000002 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843785+0000", 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0726e-05 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.239 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843819+0000", 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3463999999999997e-05 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843890+0000", 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1223999999999994e-05 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.240 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2549 4.a 4:5e6df132:::smithi15938319-47:head [write 1879985~787072] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.349138921999995, 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.036803876, 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2549 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.241 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807242+0000", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.807243+0000", 2024-09-13T13:52:37.242 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4750000000000001e-06 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808672+0000", 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00142939 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808689+0000", 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6239000000000002e-05 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.243 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808699+0000", 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0691e-05 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808760+0000", 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0529999999999998e-05 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.808769+0000", 2024-09-13T13:52:37.244 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.2839999999999995e-06 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.823985+0000", 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0152161959999999 2024-09-13T13:52:37.245 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824021+0000", 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5970999999999998e-05 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.246 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840349+0000", 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016327327999999999 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.247 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840443+0000", 2024-09-13T13:52:37.248 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4350000000000003e-05 2024-09-13T13:52:37.248 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.248 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.248 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.248 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.840585+0000", 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014192499999999999 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843898+0000", 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033129840000000002 2024-09-13T13:52:37.249 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843939+0000", 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1468999999999999e-05 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843970+0000", 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0514e-05 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.250 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844045+0000", 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.5530000000000004e-05 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2550 4.a 4:5e6df132:::smithi15938319-47:head [write 3280184~165251] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.345112001000004, 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0356691099999999, 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.251 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2550 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.252 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811268+0000", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811270+0000", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0699999999999999e-06 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811553+0000", 2024-09-13T13:52:37.253 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00028316499999999999 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811564+0000", 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1134e-05 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811573+0000", 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4180000000000003e-06 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.254 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811617+0000", 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3727000000000003e-05 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811623+0000", 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4309999999999999e-06 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824067+0000", 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.012444095 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.255 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824149+0000", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1351000000000001e-05 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.841975+0000", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.017826092000000002 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842025+0000", 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9895999999999998e-05 2024-09-13T13:52:37.256 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.842113+0000", 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8048999999999997e-05 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844149+0000", 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020358020000000002 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.257 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846823+0000", 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0026746510000000001 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846863+0000", 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9988999999999999e-05 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846938+0000", 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4239999999999994e-05 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.258 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2551 4.a 4:5e6df132:::smithi15938319-47:head [setxattr _header (57) in=7b,truncate 3445435] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.344787045000004, 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035512663, 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2551 2024-09-13T13:52:37.259 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811593+0000", 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.260 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811594+0000", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811598+0000", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.941e-06 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811606+0000", 2024-09-13T13:52:37.261 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3089999999999996e-06 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811613+0000", 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811640+0000", 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7729000000000002e-05 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811645+0000", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1779999999999996e-06 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824209+0000", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125648199999999 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824246+0000", 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6146000000000003e-05 2024-09-13T13:52:37.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843102+0000", 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.018856210000000002 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843182+0000", 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0117999999999994e-05 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.264 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843327+0000", 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000144738 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.844724+0000", 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001397705 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.846975+0000", 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.002250971 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.265 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847010+0000", 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4288000000000001e-05 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847106+0000", 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.6392999999999997e-05 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.266 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2552 4.a 4:5e6df132:::smithi15938319-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 95.344755520000007, 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.035662613, 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2552 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811625+0000", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811626+0000", 2024-09-13T13:52:37.268 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9999999999999997e-07 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811628+0000", 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9510000000000002e-06 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811635+0000", 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8179999999999998e-06 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.269 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811642+0000", 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3199999999999996e-06 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811698+0000", 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6249000000000003e-05 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:01.811706+0000", 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5019999999999992e-06 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.270 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824277+0000", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0125704879999999 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.824308+0000", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1047999999999998e-05 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843688+0000", 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.019380541000000001 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.271 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.843733+0000", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4728999999999998e-05 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847133+0000", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033999759999999999 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847188+0000", 2024-09-13T13:52:37.272 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4592999999999997e-05 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.847288+0000", 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.9797999999999996e-05 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPushReply(4.2 88/86 [PushReplyOp(4:4d382039:::smithi15938319-28:head)])", 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:37.273 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 92.317065518999996, 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.58005522499999995, 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.274 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839315+0000", 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839316+0000", 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3209999999999999e-06 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:04.839324+0000", 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0369999999999995e-06 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.275 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418437+0000", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.57911305199999996 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418447+0000", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.8970000000000003e-06 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.418529+0000", 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1606000000000004e-05 2024-09-13T13:52:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:05.419370+0000", 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00084131199999999998 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:3255 4.a 4:5004af91:::smithi15938319-34:head [append 0~0] snapc 23=[23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 68.408482661999997, 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.285444431, 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.278 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 3255 2024-09-13T13:52:37.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.281 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747898+0000", 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.39e-07 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747900+0000", 2024-09-13T13:52:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1220000000000002e-06 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747906+0000", 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5979999999999996e-06 2024-09-13T13:52:37.283 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.747911+0000", 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4e-06 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.284 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750840+0000", 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029289250000000002 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.285 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.750936+0000", 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5449e-05 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751081+0000", 2024-09-13T13:52:37.286 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014517800000000001 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.751499+0000", 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00041844799999999998 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033276+0000", 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28177714300000001 2024-09-13T13:52:37.287 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033312+0000", 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5370999999999997e-05 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033342+0000", 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0457999999999999e-05 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.288 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:3255 4.a e112/85 ondisk, result = 0)", 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 68.403843567999999, 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.280808787, 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:37.289 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752537+0000", 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3599999999999999e-07 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.290 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:28.752540+0000", 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9849999999999998e-06 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033199+0000", 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.28065851600000002 2024-09-13T13:52:37.291 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033203+0000", 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5029999999999999e-06 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033267+0000", 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3596999999999994e-05 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.292 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033273+0000", 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.789e-06 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:29.033346+0000", 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2960999999999995e-05 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.293 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:37.295 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:37.295 INFO:tasks.rados.rados.0.smithi159.stdout:2038: expect (ObjNum 157 snap 13 seq_num 157) 2024-09-13T13:52:37.295 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:37.296 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:37.293+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:37.362 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 2519959 2024-09-13T13:52:37.396 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:37.395+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:37.404 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:37.404 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:37.404 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:37.404 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:37.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:2040: finishing set_chunk to oid 14 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 14 set_chunk returned error code -95 offset: 443392 length: 2330112 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 14 v 0 (ObjNum 315 snap 45 seq_num 315) dirty exists 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:2041: finishing write tid 1 to smithi15938319-36 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:2041: finishing write tid 2 to smithi15938319-36 2024-09-13T13:52:37.414 INFO:tasks.rados.rados.0.smithi159.stdout:2041: finishing write tid 3 to smithi15938319-36 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2029: done (14 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2030: done (13 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2031: done (12 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2032: done (11 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2033: done (10 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2034: done (9 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2035: done (8 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2036: done (7 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2037: done (6 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2038: done (5 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2039: done (4 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2040: done (3 left) 2024-09-13T13:52:37.415 INFO:tasks.rados.rados.0.smithi159.stdout:2045: set_chunk oid 47 target oid 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2045: set_chunk oid 47 offset: 1834496 length: 198656 target oid 29c77c1fc57381ab3123c7bdbb7b45d712d3a8800c98890b8c35c6d0080fc11f offset: 0 snap 54 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2046: write (excl) oid 40 current snap is 65 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2041: finishing write tid 4 to smithi15938319-36 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2041: oid 36 updating version 0 to 325 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2041: oid 36 version 325 is already newer than 324 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2041: oid 36 version 325 is already newer than 323 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 36 v 325 (ObjNum 397 snap 65 seq_num 397) dirty exists 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2041: left oid 36 (ObjNum 397 snap 65 seq_num 397) 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2042: finishing write tid 1 to smithi15938319-37 2024-09-13T13:52:37.418 INFO:tasks.rados.rados.0.smithi159.stdout:2042: finishing write tid 2 to smithi15938319-37 2024-09-13T13:52:37.419 INFO:tasks.rados.rados.0.smithi159.stdout:2046: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-13T13:52:37.424 INFO:tasks.rados.rados.0.smithi159.stdout:2046: writing smithi15938319-40 from 705993 to 1180700 tid 1 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2046: writing smithi15938319-40 from 1582080 to 2194221 tid 2 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2041: done (4 left) 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2047: read oid 8 snap -1 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2047: expect (ObjNum 317 snap 46 seq_num 317) 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2048: write oid 11 current snap is 65 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: finishing write tid 3 to smithi15938319-37 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: finishing write tid 4 to smithi15938319-37 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: oid 37 updating version 0 to 271 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: oid 37 updating version 271 to 272 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: oid 37 updating version 272 to 273 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 37 v 273 (ObjNum 398 snap 65 seq_num 398) dirty exists 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2042: left oid 37 (ObjNum 398 snap 65 seq_num 398) 2024-09-13T13:52:37.430 INFO:tasks.rados.rados.0.smithi159.stdout:2044: got (22) Invalid argument 2024-09-13T13:52:37.431 INFO:tasks.rados.rados.0.smithi159.stdout:2048: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-13T13:52:37.436 INFO:tasks.rados.rados.0.smithi159.stdout:2048: writing smithi15938319-11 from 722800 to 1272756 tid 1 2024-09-13T13:52:37.444 INFO:tasks.rados.rados.0.smithi159.stdout:2048: writing smithi15938319-11 from 2040562 to 2722800 tid 2 2024-09-13T13:52:37.444 INFO:tasks.rados.rados.0.smithi159.stdout:2042: done (5 left) 2024-09-13T13:52:37.444 INFO:tasks.rados.rados.0.smithi159.stdout:2044: done (4 left) 2024-09-13T13:52:37.444 INFO:tasks.rados.rados.0.smithi159.stdout:2049: set_chunk oid 17 target oid 2024-09-13T13:52:37.453 INFO:tasks.rados.rados.0.smithi159.stdout:2049: set_chunk oid 17 offset: 1007104 length: 926208 target oid f866d780ba6bd4bd55da6d6463fa4b3ea38f3fd75974ceebfe27788354dbafb0 offset: 0 snap -1 2024-09-13T13:52:37.453 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 5 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2045: finishing set_chunk to oid 47 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2045: done (4 left) 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 4 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: finishing write tid 1 to smithi15938319-40 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: finishing write tid 2 to smithi15938319-40 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: finishing write tid 3 to smithi15938319-40 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: finishing write tid 4 to smithi15938319-40 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: oid 40 updating version 0 to 306 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: oid 40 version 306 is already newer than 305 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: oid 40 updating version 306 to 307 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 40 v 307 (ObjNum 399 snap 65 seq_num 399) dirty exists 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2046: left oid 40 (ObjNum 399 snap 65 seq_num 399) 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2048: finishing write tid 1 to smithi15938319-11 2024-09-13T13:52:37.454 INFO:tasks.rados.rados.0.smithi159.stdout:2047: expect (ObjNum 317 snap 46 seq_num 317) 2024-09-13T13:52:37.497 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:37.496+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:37.498 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 3255935 2024-09-13T13:52:37.503 INFO:tasks.rados.rados.0.smithi159.stdout:seeking to 0 2024-09-13T13:52:37.522 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:37.522 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:37.522 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:37.522 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:37.522 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:37.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:37.599 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:37.598+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: finishing write tid 2 to smithi15938319-11 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: finishing write tid 3 to smithi15938319-11 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: finishing write tid 4 to smithi15938319-11 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: oid 11 updating version 0 to 311 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: oid 11 version 311 is already newer than 309 2024-09-13T13:52:37.634 INFO:tasks.rados.rados.0.smithi159.stdout:2048: oid 11 version 311 is already newer than 310 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 11 v 311 (ObjNum 400 snap 65 seq_num 400) dirty exists 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:2048: left oid 11 (ObjNum 400 snap 65 seq_num 400) 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:2046: done (3 left) 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:2047: done (2 left) 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:2048: done (1 left) 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout: waiting on 1 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:2049: finishing set_chunk to oid 17 2024-09-13T13:52:37.635 INFO:tasks.rados.rados.0.smithi159.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 1007104 length: 926208 2024-09-13T13:52:37.636 INFO:tasks.rados.rados.0.smithi159.stdout:update_object_version oid 17 v 0 (ObjNum 369 snap 54 seq_num 369) dirty exists 2024-09-13T13:52:37.636 INFO:tasks.rados.rados.0.smithi159.stdout:2049: done (0 left) 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:255 4.d 4:b771c1ad:::smithi15938319-6:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.639 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.73888527899999, 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.016278148999999999, 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.640 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 255 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:37.641 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898154+0000", 2024-09-13T13:52:37.642 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898155+0000", 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7579999999999999e-06 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898173+0000", 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7285000000000001e-05 2024-09-13T13:52:37.643 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898186+0000", 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3492999999999999e-05 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898193+0000", 2024-09-13T13:52:37.644 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0740000000000004e-06 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898259+0000", 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5989999999999997e-05 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898359+0000", 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010012400000000001 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.645 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912254+0000", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013894131000000001 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.912876+0000", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00062274499999999996 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914359+0000", 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001482546 2024-09-13T13:52:37.646 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914394+0000", 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5645000000000002e-05 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.647 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.914432+0000", 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7357999999999997e-05 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:261 4.c 4:34c00adb:::smithi15938319-12:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 144.73881742200001, 2024-09-13T13:52:37.648 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.012989277, 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 261 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:37.649 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.650 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898222+0000", 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.651 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898232+0000", 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7079999999999992e-06 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898240+0000", 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.011e-06 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.652 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898246+0000", 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2879999999999998e-06 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898299+0000", 2024-09-13T13:52:37.653 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2363e-05 2024-09-13T13:52:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.898380+0000", 2024-09-13T13:52:37.654 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1037000000000004e-05 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.908535+0000", 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.010155022999999999 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.909735+0000", 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001200741 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.655 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911132+0000", 2024-09-13T13:52:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0013966549999999999 2024-09-13T13:52:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.656 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911166+0000", 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4178999999999999e-05 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:12.911211+0000", 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4549999999999999e-05 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.657 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.40884281000001, 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501508226, 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.658 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.659 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228196+0000", 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228197+0000", 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3170000000000001e-06 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.660 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228203+0000", 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0190000000000003e-06 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228213+0000", 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0023999999999999e-05 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.661 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228220+0000", 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.956e-06 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228318+0000", 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.7311000000000004e-05 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228328+0000", 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0146000000000001e-05 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.662 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729690+0000", 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50136237800000005 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729704+0000", 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4075e-05 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.663 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.408808335, 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50171734000000001, 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.664 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.665 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228231+0000", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1300000000000001e-07 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228235+0000", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.642e-06 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228238+0000", 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.569e-06 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.666 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228241+0000", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0180000000000002e-06 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228356+0000", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000114636 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228362+0000", 2024-09-13T13:52:37.667 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4150000000000004e-06 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729940+0000", 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157770400000001 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729948+0000", 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0430000000000008e-06 2024-09-13T13:52:37.668 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4.db50032c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.40879334900001, 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50176469999999995, 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.669 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.670 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228246+0000", 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5899999999999998e-07 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.671 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228247+0000", 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.626e-06 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228250+0000", 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9909999999999998e-06 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228254+0000", 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4120000000000001e-06 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.672 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228399+0000", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000145129 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228419+0000", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0199000000000001e-05 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.729997+0000", 2024-09-13T13:52:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50157754600000004 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730010+0000", 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3538e-05 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.674 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:37.675 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.40877929999999, 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50177167499999997, 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.676 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.677 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228260+0000", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228263+0000", 2024-09-13T13:52:37.678 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7259999999999998e-06 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228267+0000", 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3030000000000001e-06 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228270+0000", 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.8380000000000002e-06 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.679 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228465+0000", 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00019493000000000001 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.680 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228472+0000", 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2450000000000001e-06 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730024+0000", 2024-09-13T13:52:37.681 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50155224799999998 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730031+0000", 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0060000000000003e-06 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.682 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.40875758499999, 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50183147100000003, 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.683 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228281+0000", 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.684 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228282+0000", 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.84e-07 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5489999999999999e-06 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228288+0000", 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7369999999999997e-06 2024-09-13T13:52:37.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228293+0000", 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2179999999999998e-06 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228529+0000", 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00023630799999999999 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.686 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228539+0000", 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5210000000000008e-06 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730106+0000", 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.501566807 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730113+0000", 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0469999999999996e-06 2024-09-13T13:52:37.687 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4.c45c97bc (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.40872573999999, 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50182354600000001, 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.688 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:37.689 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228313+0000", 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6399999999999998e-07 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228315+0000", 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6500000000000001e-06 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.690 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228319+0000", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8809999999999998e-06 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0010000000000002e-06 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228574+0000", 2024-09-13T13:52:37.691 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00025166400000000002 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.228581+0000", 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1180000000000003e-06 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730131+0000", 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.50154984300000005 2024-09-13T13:52:37.692 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730137+0000", 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.1249999999999998e-06 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.693 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1230 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 0~1640377,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.24702901399999, 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34068805800000002, 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.694 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1230 2024-09-13T13:52:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.695 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.696 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390010+0000", 2024-09-13T13:52:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.697 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390011+0000", 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1100000000000005e-07 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390015+0000", 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0640000000000004e-06 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390022+0000", 2024-09-13T13:52:37.698 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6009999999999997e-06 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390028+0000", 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4539999999999998e-06 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4845000000000001e-05 2024-09-13T13:52:37.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7209999999999997e-06 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730190+0000", 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34010040800000002 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.700 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730355+0000", 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016552200000000001 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730698+0000", 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000342632 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.701 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1231 4.c 4:34c00adb:::smithi15938319-12:head [sparse-read 1978750~575035] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.24699586400001, 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34087025100000001, 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.702 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1231 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390043+0000", 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.703 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390044+0000", 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390046+0000", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390054+0000", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2029999999999992e-06 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390059+0000", 2024-09-13T13:52:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6240000000000001e-06 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390119+0000", 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0071000000000001e-05 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390132+0000", 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3135999999999999e-05 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.706 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730719+0000", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34058665999999999 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730780+0000", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0819999999999997e-05 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.730913+0000", 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00013379599999999999 2024-09-13T13:52:37.707 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1232 4.c 4:34c00adb:::smithi15938319-12:head [read 0~3260525,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.24696564600001, 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34316925799999998, 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.708 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1232 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390073+0000", 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.709 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390074+0000", 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8800000000000003e-07 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.710 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390077+0000", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7949999999999999e-06 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390083+0000", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390089+0000", 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9549999999999997e-06 2024-09-13T13:52:37.711 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390154+0000", 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5189000000000002e-05 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390158+0000", 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1590000000000003e-06 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.712 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731077+0000", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34091924800000001 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.731146+0000", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8986000000000007e-05 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733243+0000", 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020963399999999999 2024-09-13T13:52:37.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1233 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.246923297, 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35798184199999999, 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.714 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1233 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.715 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390116+0000", 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390123+0000", 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.0520000000000004e-06 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.716 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390134+0000", 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1274000000000001e-05 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390140+0000", 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8969999999999996e-06 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.717 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390190+0000", 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9385000000000002e-05 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.718 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390204+0000", 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4666e-05 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.733283+0000", 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34307847499999999 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.719 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734242+0000", 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00095945999999999998 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747432+0000", 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013189612999999999 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.747498+0000", 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6178000000000002e-05 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.720 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748097+0000", 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00059934599999999995 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1234 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 0~937075] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.24688907999999, 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35877602400000003, 2024-09-13T13:52:37.721 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1234 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.722 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390150+0000", 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.723 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390151+0000", 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6199999999999997e-07 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390153+0000", 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6359999999999998e-06 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.724 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390159+0000", 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7309999999999998e-06 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390173+0000", 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3793e-05 2024-09-13T13:52:37.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390235+0000", 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2615999999999998e-05 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390239+0000", 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8909999999999997e-06 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.726 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734292+0000", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34405234000000001 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734368+0000", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6483000000000006e-05 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748140+0000", 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013771867 2024-09-13T13:52:37.727 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748196+0000", 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5667000000000001e-05 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748926+0000", 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00073033799999999997 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.728 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:1235 4.c 4:3de93a23:::smithi15938319-15:head [sparse-read 1459897~166818] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 129.24685580900001, 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.35905498600000002, 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 1235 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.729 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390183+0000", 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.730 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390184+0000", 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.89e-07 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390186+0000", 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6690000000000002e-06 2024-09-13T13:52:37.731 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390193+0000", 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5339999999999996e-06 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390200+0000", 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.6569999999999997e-06 2024-09-13T13:52:37.732 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390295+0000", 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4408999999999997e-05 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.733 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for peered", 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.390303+0000", 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8520000000000004e-06 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.734 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734408+0000", 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.34410492399999998 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:37.735 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.734471+0000", 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3501000000000001e-05 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748971+0000", 2024-09-13T13:52:37.736 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014499997000000001 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749062+0000", 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1197000000000001e-05 2024-09-13T13:52:37.737 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.738 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.738 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.738 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.749238+0000", 2024-09-13T13:52:37.738 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00017575699999999999 2024-09-13T13:52:37.738 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGRecoveryDeleteReply(4.c e53,52 4:358f784f:::smithi15938319-41:head,51'46)", 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:37.739 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 128.84457257599999, 2024-09-13T13:52:37.740 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99557941599999999, 2024-09-13T13:52:37.740 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.740 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.740 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:37.740 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.741 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792466+0000", 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792467+0000", 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9700000000000003e-07 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.743 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.792472+0000", 2024-09-13T13:52:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9099999999999996e-06 2024-09-13T13:52:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.744 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787553+0000", 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.99508121900000002 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787563+0000", 2024-09-13T13:52:37.745 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0179e-05 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.787864+0000", 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00030082399999999999 2024-09-13T13:52:37.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.747 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.747 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:29.788046+0000", 2024-09-13T13:52:37.747 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00018198699999999999 2024-09-13T13:52:37.747 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2586 4.c 4:34327e96:::smithi15938319-44:head [write 729470~543821,stat] snapc 17=[17,16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-13T13:52:37.748 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 94.819505966999998, 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.015347322, 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.749 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.750 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.750 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.750 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2586 2024-09-13T13:52:37.750 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.750 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.751 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817533+0000", 2024-09-13T13:52:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.752 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.817535+0000", 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7489999999999999e-06 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.753 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818291+0000", 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00075675899999999997 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.754 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818307+0000", 2024-09-13T13:52:37.755 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.5486e-05 2024-09-13T13:52:37.755 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.755 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.755 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.755 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818318+0000", 2024-09-13T13:52:37.756 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1534e-05 2024-09-13T13:52:37.756 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.756 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.756 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.756 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818375+0000", 2024-09-13T13:52:37.757 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.6378000000000001e-05 2024-09-13T13:52:37.757 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.757 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.757 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:37.757 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.818381+0000", 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4740000000000002e-06 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822734+0000", 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043530299999999999 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.758 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822834+0000", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010008799999999999 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.822977+0000", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014281799999999999 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.826424+0000", 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034465670000000002 2024-09-13T13:52:37.759 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832807+0000", 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0063826999999999998 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832841+0000", 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4650000000000002e-05 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.760 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:02.832880+0000", 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9088999999999998e-05 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:2965 4.c 4:358f784f:::smithi15938319-41:head [append 0~0] snapc 1d=[1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 80.763427444000001, 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.014521253, 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.761 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 2965 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873611+0000", 2024-09-13T13:52:37.762 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873612+0000", 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2999999999999999e-07 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873613+0000", 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.398e-06 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.763 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873616+0000", 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7190000000000001e-06 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873618+0000", 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6170000000000001e-06 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873745+0000", 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000126389 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.764 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.873747+0000", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4030000000000001e-06 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882806+0000", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0090587889999999994 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882861+0000", 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4793999999999997e-05 2024-09-13T13:52:37.765 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.882955+0000", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.4377999999999997e-05 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.883725+0000", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00076983499999999996 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888075+0000", 2024-09-13T13:52:37.766 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0043496630000000001 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888105+0000", 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0735000000000003e-05 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:16.888133+0000", 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7302999999999999e-05 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.767 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop_reply(client.4571.0:4008 4.c e138/52 ondisk, result = 0)", 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 37.547256228000002, 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95524980500000001, 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:37.768 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089783+0000", 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2799999999999996e-07 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.769 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:00.089787+0000", 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.422e-06 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044661+0000", 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.95487436400000003 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044676+0000", 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4746000000000001e-05 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.770 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044804+0000", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012786699999999999 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.044823+0000", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9296e-05 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:01.045033+0000", 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00020948200000000001 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.771 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4104 4.c 4:358f784f:::smithi15938319-41:head [write 463959~478445,stat] snapc 30=[30,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 34.551219846000002, 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013243293, 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.772 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4104 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085819+0000", 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.773 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.085821+0000", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.199e-06 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086649+0000", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00082724800000000005 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086672+0000", 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3768e-05 2024-09-13T13:52:37.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086684+0000", 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1584e-05 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086762+0000", 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8566000000000006e-05 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:37.775 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.086770+0000", 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7800000000000001e-06 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091302+0000", 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0045321670000000001 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091427+0000", 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00012461500000000001 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.776 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.091571+0000", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00014418699999999999 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098930+0000", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0073584499999999999 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.098994+0000", 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.4639000000000002e-05 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.777 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099030+0000", 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5627000000000003e-05 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:03.099062+0000", 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.778 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_op(client.4571.0:4618 4.c 4:3de93a23:::smithi15938319-15:head [write 470695~788515] snapc 3d=[3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,2d,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:52:28.206843+0000", 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 9.4301958769999992, 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.013353291999999999, 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "client_info": { 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "client": "client.4571", 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "tid": 4618 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:37.779 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.206843+0000", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.206843+0000", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.206845+0000", 2024-09-13T13:52:37.780 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7069999999999999e-06 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.207766+0000", 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000920997 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.207785+0000", 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9307e-05 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:37.781 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.207793+0000", 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.8539999999999997e-06 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.207862+0000", 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9449000000000005e-05 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op must wait for map", 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.207870+0000", 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.4970000000000004e-06 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.782 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.212050+0000", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041796660000000003 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.212154+0000", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00010420300000000001 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.212352+0000", 2024-09-13T13:52:37.783 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000198672 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "op_commit", 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.216600+0000", 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0042471389999999996 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.220074+0000", 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034743280000000001 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.784 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.220134+0000", 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.9923000000000002e-05 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:52:28.220196+0000", 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.2550000000000003e-05 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:37.785 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:37.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:37.787 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:37.786+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:37.887 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:37.887+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:37.902 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:37.902 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:37.903 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:37.903 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:37.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:37.989 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:37.988+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:38.015 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:38.015 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:38.015 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:38.015 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:38.015 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:38.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:38.090 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:38.089+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:38.130 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:38.130 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:38.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:38.130 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:38.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.131 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:268 4.7 4:e1bd516b:::smithi15938319-19:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:38.131 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:38.131 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 145.21224367299999, 2024-09-13T13:52:38.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013265704999999999, 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 268 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915291+0000", 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915292+0000", 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915301+0000", 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915316+0000", 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5957e-05 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915327+0000", 2024-09-13T13:52:38.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0736e-05 2024-09-13T13:52:38.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915383+0000", 2024-09-13T13:52:38.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000001e-05 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.915460+0000", 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6858000000000002e-05 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.925954+0000", 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010494101 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.926603+0000", 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00064925300000000003 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928490+0000", 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018860700000000001 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928519+0000", 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9459999999999999e-05 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.928556+0000", 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:274 4.7 4:e26a40dd:::smithi15938319-25:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 145.20744414399999, 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012001847, 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.142 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 274 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920090+0000", 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920092+0000", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.04e-06 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920102+0000", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.025e-05 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920129+0000", 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6109e-05 2024-09-13T13:52:38.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920143+0000", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4676e-05 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920258+0000", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000114694 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.920323+0000", 2024-09-13T13:52:38.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4916e-05 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.930378+0000", 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010054745 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.931070+0000", 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069260199999999995 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932002+0000", 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00093131900000000001 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932053+0000", 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1564000000000003e-05 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:12.932092+0000", 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8931999999999999e-05 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.147 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:300 4.7 4:e63fe140:::smithi15938319-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 144.358096106, 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097442429999999997, 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.148 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 300 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769438+0000", 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769440+0000", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5400000000000001e-06 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769458+0000", 2024-09-13T13:52:38.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.324e-05 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0010000000000001e-06 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769522+0000", 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5297999999999998e-05 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769585+0000", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778013+0000", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008427861 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778390+0000", 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000377282 2024-09-13T13:52:38.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779082+0000", 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00069211900000000005 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779138+0000", 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6326000000000002e-05 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.779182+0000", 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4075000000000001e-05 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1260 4.7 4:e1bd516b:::smithi15938319-19:head [write 695823~777044,stat] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.853798506, 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79460949800000003, 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.154 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1260 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273736+0000", 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.273738+0000", 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3439999999999999e-06 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274521+0000", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00078313000000000005 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274542+0000", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0319999999999999e-05 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274554+0000", 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2777e-05 2024-09-13T13:52:38.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274634+0000", 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9351000000000006e-05 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.274644+0000", 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0505e-05 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.052414+0000", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77776927600000001 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.053841+0000", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001427191 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062333+0000", 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084923889999999995 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062403+0000", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.062552+0000", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000148899 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068196+0000", 2024-09-13T13:52:38.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056445669999999996 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068209+0000", 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2815e-05 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068247+0000", 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8426000000000001e-05 2024-09-13T13:52:38.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068345+0000", 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7930000000000006e-05 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1261 4.7 4:e1bd516b:::smithi15938319-19:head [write 1925015~639682] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.84793855699999, 2024-09-13T13:52:38.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79154317699999999, 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1261 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279596+0000", 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.279597+0000", 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.35e-06 2024-09-13T13:52:38.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280220+0000", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00062313699999999995 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280234+0000", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3764999999999999e-05 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280242+0000", 2024-09-13T13:52:38.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1389999999999995e-06 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280342+0000", 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100079 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.280351+0000", 2024-09-13T13:52:38.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5529999999999993e-06 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054030+0000", 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.77367957399999998 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054069+0000", 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8509e-05 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064145+0000", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010075842 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064216+0000", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1423999999999999e-05 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.064343+0000", 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012723500000000001 2024-09-13T13:52:38.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068579+0000", 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042351749999999999 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071027+0000", 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002448048 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071064+0000", 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.752e-05 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071139+0000", 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4826999999999994e-05 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1262 4.7 4:e1bd516b:::smithi15938319-19:head [write 3328058~167765] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.170 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.84299391600001, 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78725081500000005, 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1262 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284540+0000", 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284541+0000", 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5900000000000005e-07 2024-09-13T13:52:38.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284718+0000", 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017633899999999999 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284728+0000", 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0071999999999999e-05 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284736+0000", 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6230000000000003e-06 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284807+0000", 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0597000000000002e-05 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284813+0000", 2024-09-13T13:52:38.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2090000000000002e-06 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054114+0000", 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76930088699999999 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054146+0000", 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1480999999999999e-05 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065345+0000", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011199185 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065415+0000", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0454e-05 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.065514+0000", 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9222999999999995e-05 2024-09-13T13:52:38.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070963+0000", 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054489860000000003 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071678+0000", 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071460299999999996 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071710+0000", 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2462999999999997e-05 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071791+0000", 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0734000000000003e-05 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1263 4.7 4:e1bd516b:::smithi15938319-19:head [setxattr _header (56) in=7b,truncate 3495823] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.178 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.842783338, 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78732732000000005, 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1263 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284751+0000", 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284752+0000", 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3900000000000004e-07 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284774+0000", 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2093e-05 2024-09-13T13:52:38.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284782+0000", 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9729999999999997e-06 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284788+0000", 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1e-06 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284842+0000", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3866000000000002e-05 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284849+0000", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8639999999999998e-06 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054174+0000", 2024-09-13T13:52:38.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76932569399999995 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054207+0000", 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3238999999999998e-05 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067659+0000", 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013452007 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067765+0000", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105498 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.067926+0000", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016110699999999999 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.070967+0000", 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030410609999999999 2024-09-13T13:52:38.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071963+0000", 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00099541199999999999 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.071997+0000", 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4833999999999999e-05 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072078+0000", 2024-09-13T13:52:38.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0933000000000006e-05 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.188 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1264 4.7 4:e1bd516b:::smithi15938319-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.189 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:38.189 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.84273523900001, 2024-09-13T13:52:38.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.78747845000000005, 2024-09-13T13:52:38.189 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.189 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1264 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.190 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284799+0000", 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284800+0000", 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8599999999999998e-07 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284802+0000", 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8289999999999999e-06 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284809+0000", 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8669999999999996e-06 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284815+0000", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0970000000000002e-06 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284870+0000", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4672000000000001e-05 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.284874+0000", 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:38.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054326+0000", 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.76945145699999995 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054364+0000", 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8472e-05 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068394+0000", 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014029102999999999 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.068454+0000", 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0643000000000002e-05 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072113+0000", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036588179999999999 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072167+0000", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4146e-05 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.072278+0000", 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000110386 2024-09-13T13:52:38.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1271 4.7 4:e63fe140:::smithi15938319-1:5 [read 1323387~693420,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.75965482800001, 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68744168100000003, 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1271 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367879+0000", 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367882+0000", 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1940000000000001e-06 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367893+0000", 2024-09-13T13:52:38.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1279e-05 2024-09-13T13:52:38.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367923+0000", 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833e-05 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367943+0000", 2024-09-13T13:52:38.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0069e-05 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368143+0000", 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000200447 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368152+0000", 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7560000000000005e-06 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054406+0000", 2024-09-13T13:52:38.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68625395300000003 2024-09-13T13:52:38.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.054595+0000", 2024-09-13T13:52:38.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018912599999999999 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055321+0000", 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072602400000000003 2024-09-13T13:52:38.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.206 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.206 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.206 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1272 4.7 4:e63fe140:::smithi15938319-1:5 [read 233422~1783385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.75954611700001, 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68851057299999996, 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.207 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1272 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.208 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367988+0000", 2024-09-13T13:52:38.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367990+0000", 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.376e-06 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.367998+0000", 2024-09-13T13:52:38.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2339999999999994e-06 2024-09-13T13:52:38.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368015+0000", 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6983e-05 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368030+0000", 2024-09-13T13:52:38.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5186e-05 2024-09-13T13:52:38.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368178+0000", 2024-09-13T13:52:38.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014779100000000001 2024-09-13T13:52:38.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368186+0000", 2024-09-13T13:52:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1089999999999998e-06 2024-09-13T13:52:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055360+0000", 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.687174336 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.055459+0000", 2024-09-13T13:52:38.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8981999999999995e-05 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056499+0000", 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001039576 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.218 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1273 4.7 4:e63fe140:::smithi15938319-1:5 [sparse-read 860163~545468] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 129.759430837, 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68867034999999999, 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.219 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1273 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368103+0000", 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368105+0000", 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3370000000000001e-06 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368114+0000", 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0529999999999996e-06 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368144+0000", 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0105999999999999e-05 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368150+0000", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7849999999999997e-06 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368205+0000", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4826e-05 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:28.368209+0000", 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0169999999999998e-06 2024-09-13T13:52:38.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056532+0000", 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.68832325500000002 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056606+0000", 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:29.056774+0000", 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016772999999999999 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2580 4.7 4:e1bd516b:::smithi15938319-19:head [write 712663~468572] snapc 16=[16,15,14,13,12,f] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 96.251460902999995, 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010180348000000001, 2024-09-13T13:52:38.225 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.227 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.227 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2580 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876073+0000", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876075+0000", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7969999999999999e-06 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876880+0000", 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00080466200000000004 2024-09-13T13:52:38.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876896+0000", 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6412000000000001e-05 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.876906+0000", 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9580000000000002e-06 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877008+0000", 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101561 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.877014+0000", 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0460000000000002e-06 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881361+0000", 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004347409 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881435+0000", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3693000000000001e-05 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.881564+0000", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012924200000000001 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886162+0000", 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045976109999999997 2024-09-13T13:52:38.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886176+0000", 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.411e-05 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886216+0000", 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0337999999999999e-05 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:01.886254+0000", 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(osd.1.0:130 4.7 e130/52 ondisk, result = 0)", 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.313243464000003, 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18760375600000001, 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.234 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814291+0000", 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814292+0000", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7999999999999995e-07 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:51.814296+0000", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8339999999999998e-06 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001621+0000", 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.18732465400000001 2024-09-13T13:52:38.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001634+0000", 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2883e-05 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001774+0000", 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000140104 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001805+0000", 2024-09-13T13:52:38.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1145000000000001e-05 2024-09-13T13:52:38.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:51:52.001895+0000", 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9456000000000005e-05 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4040 4.7 4:e26a40dd:::smithi15938319-25:head [write 763117~710598] snapc 30=[30,2f,2d,2a,28,27,25,24,23,22,1a,14] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 36.999515617, 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010818598, 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4040 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128019+0000", 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128020+0000", 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.778e-06 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128710+0000", 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000689637 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128727+0000", 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6628000000000001e-05 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128734+0000", 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128785+0000", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0828999999999997e-05 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.128789+0000", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1409999999999998e-06 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133000+0000", 2024-09-13T13:52:38.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042116080000000004 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133067+0000", 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6644999999999996e-05 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.133188+0000", 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121249 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138729+0000", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055408610000000002 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138749+0000", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0029000000000001e-05 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138796+0000", 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6681999999999998e-05 2024-09-13T13:52:38.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:01.138837+0000", 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1440000000000003e-05 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4225 4.7 4:e26a40dd:::smithi15938319-25:23 [read 0~3630279,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 30.802783375000001, 2024-09-13T13:52:38.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010120242, 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4225 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:38.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324751+0000", 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8000000000000001e-07 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324754+0000", 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6419999999999999e-06 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324758+0000", 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6629999999999999e-06 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324761+0000", 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.145e-06 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324877+0000", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116599 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324884+0000", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4520000000000002e-06 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329877+0000", 2024-09-13T13:52:38.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049932090000000002 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.329973+0000", 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5606000000000006e-05 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334871+0000", 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048985460000000002 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.253 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4226 4.7 4:e26a40dd:::smithi15938319-25:23 [sparse-read 844183~2211810] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e144)", 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 30.802752100999999, 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011467577, 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.254 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4226 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324782+0000", 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324783+0000", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0699999999999998e-07 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324784+0000", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.731e-06 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324789+0000", 2024-09-13T13:52:38.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324796+0000", 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.537e-06 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324912+0000", 2024-09-13T13:52:38.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000115145 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.324918+0000", 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6100000000000002e-06 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334923+0000", 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010004569 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.334990+0000", 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7143000000000004e-05 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:07.336250+0000", 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0012599320000000001 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.259 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4417 4.7 4:e26a40dd:::smithi15938319-25:head [write 484134~708729,stat] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.212065611, 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013723770999999999, 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4417 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.260 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915469+0000", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.915473+0000", 2024-09-13T13:52:38.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.827e-06 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916562+0000", 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089575 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916582+0000", 2024-09-13T13:52:38.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9825000000000001e-05 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916590+0000", 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.038e-06 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916652+0000", 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1941999999999997e-05 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.916657+0000", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921371+0000", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047142629999999998 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921469+0000", 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7625000000000001e-05 2024-09-13T13:52:38.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.921620+0000", 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015109999999999999 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928797+0000", 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071769199999999998 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929103+0000", 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030604200000000002 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929152+0000", 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8565e-05 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929192+0000", 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0849000000000001e-05 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.266 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4418 4.7 4:e26a40dd:::smithi15938319-25:head [write 1883658~200476] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.207254339999999, 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014278255, 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4418 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920280+0000", 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920281+0000", 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.24e-06 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920579+0000", 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00029755100000000001 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920594+0000", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4951e-05 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920601+0000", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8950000000000001e-06 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920658+0000", 2024-09-13T13:52:38.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7017e-05 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920666+0000", 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4370000000000007e-06 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924237+0000", 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035709560000000001 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924325+0000", 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8212000000000002e-05 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.924457+0000", 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000131686 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.929201+0000", 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047441380000000002 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934475+0000", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052734549999999998 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934520+0000", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5423999999999997e-05 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.934558+0000", 2024-09-13T13:52:38.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8293000000000001e-05 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4419 4.7 4:e26a40dd:::smithi15938319-25:head [setxattr _header (57) in=7b,truncate 2084134] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.206917888, 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014942327, 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4419 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920616+0000", 2024-09-13T13:52:38.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920617+0000", 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1800000000000005e-07 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920621+0000", 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9559999999999999e-06 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920630+0000", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3200000000000006e-06 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920637+0000", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2129999999999996e-06 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920708+0000", 2024-09-13T13:52:38.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1546999999999998e-05 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920715+0000", 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3579999999999999e-06 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.927744+0000", 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070291900000000003 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.927853+0000", 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109227 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 6", 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928066+0000", 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021321400000000001 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.933043+0000", 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049770439999999999 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935446+0000", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002402894 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935481+0000", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4431000000000003e-05 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935559+0000", 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8214999999999996e-05 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.280 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:4420 4.7 4:e26a40dd:::smithi15938319-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.206883135999998, 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01508695, 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4420 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920651+0000", 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920652+0000", 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8400000000000004e-07 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920655+0000", 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0469999999999998e-06 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920661+0000", 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920667+0000", 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1880000000000001e-06 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920742+0000", 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5790000000000005e-05 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.920746+0000", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4819999999999999e-06 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928833+0000", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0080870690000000005 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.928895+0000", 2024-09-13T13:52:38.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2094999999999994e-05 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935599+0000", 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00670383 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935659+0000", 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9737000000000001e-05 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:16.935738+0000", 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9555000000000007e-05 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.287 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:38.288 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:52:38.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:38.290 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:38.287+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:38.389 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:38.388+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:38.433 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:38.433 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:38.433 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-13T13:52:38.433 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:38.442 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:38.491 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:38.490+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:38.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:38.544 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-13T13:52:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-13T13:52:38.544 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-13T13:52:38.544 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:38.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:38.597 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:38.596+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.657 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:298 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 144.88616649599999, 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.304225146, 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 298 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769435+0000", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769436+0000", 2024-09-13T13:52:38.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3960000000000001e-06 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769445+0000", 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3389999999999993e-06 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769460+0000", 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769466+0000", 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7340000000000002e-06 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769521+0000", 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4894e-05 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.769583+0000", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1383999999999994e-05 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777520+0000", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079377990000000006 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.777986+0000", 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000465208 2024-09-13T13:52:38.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073566+0000", 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29558055100000002 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073620+0000", 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3579000000000003e-05 2024-09-13T13:52:38.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073660+0000", 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop_reply(client.4571.0:298 4.0 e42/34 ondisk, result = 0)", 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 144.87713606400001, 2024-09-13T13:52:38.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29520191800000001, 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778465+0000", 2024-09-13T13:52:38.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778466+0000", 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.44e-07 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:13.778470+0000", 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5860000000000001e-06 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073501+0000", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.29503094699999999 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073506+0000", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9860000000000002e-06 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073552+0000", 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6832000000000002e-05 2024-09-13T13:52:38.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073561+0000", 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5250000000000005e-06 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:14.073667+0000", 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106498 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.6.0:176 4.4 e45/34 4:21c00763:::smithi15938319-21:head v 45'16, mlcod=42'14)", 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.22513018800001, 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075780049999999996, 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.430471+0000", 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.73e-07 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433483+0000", 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003011176 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433530+0000", 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7312999999999999e-05 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433537+0000", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1600000000000001e-06 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433597+0000", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9935000000000001e-05 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433614+0000", 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6824999999999999e-05 2024-09-13T13:52:38.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438012+0000", 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043979120000000003 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.438049+0000", 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7511000000000001e-05 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "pg_update_log_missing(4.4 epoch 45/34 rep_tid 177 entries 45'17 (0'0) error 4:21c00763:::smithi15938319-21:head by client.4571.0:832 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 42'14)", 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 140.22209620199999, 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61916211899999996, 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.674 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433505+0000", 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433506+0000", 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.58e-07 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:18.433513+0000", 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8839999999999997e-06 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052060+0000", 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.61854746599999999 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052089+0000", 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9261e-05 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052178+0000", 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9142999999999997e-05 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:19.052667+0000", 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00048900699999999996 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1555 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'45, mlcod=59'41)", 2024-09-13T13:52:38.678 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.346264084, 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0086569460000000004, 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:38.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309337+0000", 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.309340+0000", 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1830000000000001e-06 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.310113+0000", 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00077347400000000001 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315619+0000", 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055055369999999996 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315627+0000", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9419999999999995e-06 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315689+0000", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2810000000000003e-05 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315711+0000", 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1288999999999999e-05 2024-09-13T13:52:38.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317937+0000", 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022266959999999998 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.317994+0000", 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7015000000000002e-05 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1556 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'46, mlcod=59'41)", 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.340357252, 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083912359999999998, 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315244+0000", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315245+0000", 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:38.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.315603+0000", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000358494 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322087+0000", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064842379999999998 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322094+0000", 2024-09-13T13:52:38.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6329999999999999e-06 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322153+0000", 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9305999999999998e-05 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322171+0000", 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7886000000000001e-05 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323583+0000", 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001411668 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.323635+0000", 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2516000000000003e-05 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.688 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:1558 4.4 e59/52 4:21c00763:::smithi15938319-21:head v 59'48, mlcod=59'43)", 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 123.332921817, 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44956396300000001, 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322680+0000", 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322681+0000", 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0999999999999997e-07 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.322687+0000", 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771580+0000", 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.44889338099999998 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771587+0000", 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0380000000000002e-06 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771656+0000", 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8356999999999996e-05 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.771679+0000", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2923999999999999e-05 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772199+0000", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052028999999999997 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:35.772244+0000", 2024-09-13T13:52:38.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4799000000000003e-05 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1790 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.41362807500001, 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020058678, 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1790 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241973+0000", 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241975+0000", 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8989999999999999e-06 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.241986+0000", 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0899e-05 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242006+0000", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.927e-05 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242016+0000", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0088000000000001e-05 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242100+0000", 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4258000000000005e-05 2024-09-13T13:52:38.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242108+0000", 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9119999999999998e-06 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251814+0000", 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0097062780000000005 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251826+0000", 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1484000000000001e-05 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256291+0000", 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044656820000000003 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.256495+0000", 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020369 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262032+0000", 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005537218 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.700 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1791 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 0~1875138] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.41355941, 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.020624751, 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.701 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1791 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242042+0000", 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242043+0000", 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9900000000000001e-07 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242046+0000", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4450000000000001e-06 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242058+0000", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2140999999999999e-05 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242065+0000", 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242182+0000", 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000116791 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242188+0000", 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2759999999999997e-06 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251883+0000", 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0096954299999999997 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.251891+0000", 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9650000000000008e-06 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262310+0000", 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010418247 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262415+0000", 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105424 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262667+0000", 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025177700000000001 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.707 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1792 4.4 4:2fda4b5c:::smithi15938319-8:8 [sparse-read 2245847~665579] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.413509296, 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.021085208000000001, 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.708 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1792 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242092+0000", 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3799999999999996e-07 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242097+0000", 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.777e-06 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242106+0000", 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5789999999999993e-06 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242113+0000", 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7220000000000002e-06 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242258+0000", 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144895 2024-09-13T13:52:38.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.242269+0000", 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0591000000000001e-05 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252162+0000", 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0098930189999999994 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-13T13:52:38.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.252177+0000", 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5466999999999999e-05 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262700+0000", 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.010522981000000001 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.262909+0000", 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020848399999999999 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263177+0000", 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026885499999999997 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1819 4.0 4:04ca156e:::smithi15938319-49:head [write 668945~536616,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.392551935, 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073236480000000003, 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.715 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1819 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263050+0000", 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.263051+0000", 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.652e-06 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264942+0000", 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001890629 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264954+0000", 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2438e-05 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.264961+0000", 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.092e-06 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265026+0000", 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4438999999999997e-05 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265131+0000", 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105358 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.265365+0000", 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023407699999999999 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268204+0000", 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028383599999999998 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270229+0000", 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020256200000000001 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270306+0000", 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6328999999999994e-05 2024-09-13T13:52:38.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270373+0000", 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7654e-05 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1820 4.0 4:04ca156e:::smithi15938319-49:head [write 1633422~491363] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.386955732, 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094975579999999997, 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.723 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1820 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268646+0000", 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.268647+0000", 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.595e-06 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270150+0000", 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001502169 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270164+0000", 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4827000000000001e-05 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270173+0000", 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.799e-06 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270438+0000", 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026528499999999998 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270509+0000", 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0185000000000006e-05 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270653+0000", 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144783 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272497+0000", 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018431809999999999 2024-09-13T13:52:38.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278066+0000", 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055690519999999997 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278106+0000", 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0478000000000001e-05 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278143+0000", 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7203999999999998e-05 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1821 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2268945] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.38541182199999, 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0081364209999999996, 2024-09-13T13:52:38.733 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1821 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.734 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270190+0000", 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270191+0000", 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6299999999999993e-07 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270195+0000", 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1300000000000003e-06 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270203+0000", 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1280000000000008e-06 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270209+0000", 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3470000000000004e-06 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271833+0000", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016236849999999999 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.271899+0000", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6192999999999995e-05 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272018+0000", 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011845000000000001 2024-09-13T13:52:38.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.274137+0000", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021194260000000002 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278198+0000", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040609720000000004 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278258+0000", 2024-09-13T13:52:38.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9765999999999997e-05 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278326+0000", 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8361000000000005e-05 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:1822 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:38.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.385346774, 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082264799999999996, 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1822 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270255+0000", 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8800000000000002e-07 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270259+0000", 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8709999999999999e-06 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270268+0000", 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8340000000000005e-06 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.270276+0000", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2959999999999999e-06 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272191+0000", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001914317 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.272247+0000", 2024-09-13T13:52:38.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6623999999999999e-05 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278351+0000", 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061033240000000002 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278410+0000", 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9515999999999997e-05 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:41.278481+0000", 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1009999999999999e-05 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2119 4.4 4:21c00763:::smithi15938319-21:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 107.922729472, 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067660350000000001, 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.746 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2119 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732872+0000", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732875+0000", 2024-09-13T13:52:38.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2460000000000002e-06 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732885+0000", 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0857e-05 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732889+0000", 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9929999999999997e-06 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732984+0000", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4475999999999995e-05 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.732987+0000", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.737956+0000", 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049684880000000001 2024-09-13T13:52:38.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.738024+0000", 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770999999999996e-05 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.739638+0000", 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00161378 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2147 4.0 4:04ca156e:::smithi15938319-49:head [setxattr _header (57) in=7b,truncate 2840383] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 107.850833784, 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79490567199999995, 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.752 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2147 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804768+0000", 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804769+0000", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9299999999999997e-07 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804772+0000", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5889999999999999e-06 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804779+0000", 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6900000000000003e-06 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804784+0000", 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6840000000000001e-06 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805543+0000", 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000758345 2024-09-13T13:52:38.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805607+0000", 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4227999999999996e-05 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805703+0000", 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6241000000000002e-05 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.808319+0000", 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026155240000000001 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599579+0000", 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79125968700000004 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599624+0000", 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5330000000000001e-05 2024-09-13T13:52:38.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599673+0000", 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9561000000000002e-05 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2148 4.0 4:04ca156e:::smithi15938319-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 107.85079626300001, 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79505775700000003, 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-13T13:52:38.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2148 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804805+0000", 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804806+0000", 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5799999999999999e-07 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804809+0000", 2024-09-13T13:52:38.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9950000000000001e-06 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804815+0000", 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6080000000000001e-06 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.804821+0000", 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3000000000000001e-06 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805910+0000", 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001089582 2024-09-13T13:52:38.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:50.805982+0000", 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2182999999999997e-05 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599714+0000", 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.79373146100000003 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599777+0000", 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3398000000000005e-05 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.599863+0000", 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5672000000000002e-05 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2180 4.0 4:04ca156e:::smithi15938319-49:head [write 507611~777843,stat] snapc 11=[10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 106.737877508, 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090344680000000004, 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.769 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2180 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917724+0000", 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.917727+0000", 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918623+0000", 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089595099999999997 2024-09-13T13:52:38.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918643+0000", 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0173000000000002e-05 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918652+0000", 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6449999999999994e-06 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918715+0000", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3541999999999996e-05 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.918722+0000", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9779999999999999e-06 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921078+0000", 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002355893 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921150+0000", 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1816999999999994e-05 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.921286+0000", 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013644500000000001 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.924299+0000", 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030125659999999999 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926666+0000", 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023669350000000001 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926718+0000", 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2499e-05 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:51.926758+0000", 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0244000000000003e-05 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4571.0:2324 4.0 4:04ca156e:::smithi15938319-49:head [assert-version v32,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:38.777 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 102.5238111, 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091488049999999994, 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4571", 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2324 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131790+0000", 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131792+0000", 2024-09-13T13:52:38.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.77e-06 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131800+0000", 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6450000000000005e-06 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131819+0000", 2024-09-13T13:52:38.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8782000000000001e-05 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131848+0000", 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9711999999999999e-05 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.131932+0000", 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4096000000000003e-05 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.132055+0000", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122569 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.138890+0000", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068350049999999999 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.139498+0000", 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00060784700000000001 2024-09-13T13:52:38.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140847+0000", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013495720000000001 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140884+0000", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6182e-05 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:50:56.140939+0000", 2024-09-13T13:52:38.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5624999999999997e-05 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4571.0:4660 4.4 e168/85 4:2fda4b5c:::smithi15938319-8:head v 168'245, mlcod=163'239)", 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-13T13:52:32.218253+0000", 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.4373485969999997, 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.007088642, 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "continuous": false, 2024-09-13T13:52:38.784 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218253+0000", 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218253+0000", 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-13T13:52:38.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218254+0000", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2200000000000004e-07 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218260+0000", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9750000000000004e-06 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218280+0000", 2024-09-13T13:52:38.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0219000000000001e-05 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218285+0000", 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9529999999999998e-06 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218808+0000", 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00052283699999999995 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.218831+0000", 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3467000000000001e-05 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.225310+0000", 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064794049999999997 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-13T13:52:38.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-13T13:52:32.225342+0000", 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1164000000000002e-05 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-13T13:52:38.789 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-13T13:52:38.790 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:38.791 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:38.789+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:38.791 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:38.792 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:38.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:38.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:38.793 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:38.793 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:38.794 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:38.794 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:38.794 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:38.794 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:38.795 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:52:38.795 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:38.796 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:38.796 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:38.796 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:38.814 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:38.835 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:38.835 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:38.857 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:38.857 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:38.878 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:38.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:38.891 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:38.889+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:38.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:38.920 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:38.920 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:38.935 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:38.935 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:38.941 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:38.941 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:38.944 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:38.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:38.946+0000 7fad84f5d640 0 -- 172.21.15.27:0/1610875688 >> v2:172.21.15.159:6805/4073091144 conn(0x7fad80226750 msgr2=0x7fad80228b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:38.969 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:38.969 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:38.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:38.992 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:38.991+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:39.046 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.046 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.054 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:39.086 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:39.085+0000 7f4ffbd7a640 -1 osd.1 172 handle_osd_map injecting map crc failure 2024-09-13T13:52:39.092 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:39.091+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:39.158 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.158 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:39.207 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:39.206+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:39.294 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:39.293+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:39.309 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.309 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:39.395 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:39.395+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:39.462 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.462 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.471 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:39.497 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:39.496+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:39.573 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.573 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:39.626 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:39.625+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:39.696 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.696 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.698 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:39.698+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:39.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:39.799 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:39.798+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:39.849 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.849 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:39.900 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:39.899+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:39.961 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:39.961 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:39.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:40.014 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:40.013+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:40.073 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.073 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:40.103 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:40.101+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:40.204 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:40.203+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:40.224 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.225 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:40.304 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:40.303+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:40.336 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.336 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:40.405 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:40.403+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:40.447 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.447 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.456 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:40.506 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:40.505+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:40.557 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.557 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.565 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:40.610 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:40.609+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:40.669 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.669 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:40.708 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:40.707+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:40.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:40.809 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:40.808+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:40.821 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.821 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:40.910 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:40.908+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:40.973 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:40.973 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:40.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:40.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:41.025 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:41.024+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:41.084 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:41.084 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:41.093 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:41.107 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:41.107 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:41.112 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:41.110+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:41.130 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:41.130 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:41.151 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:41.172 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:41.173 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:41.173 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:41.194 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:41.209 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.209 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:41.209 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:41.209 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:41.212 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:41.210+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:41.215 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:41.215 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:41.218 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:41.229 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:41.237 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:41.237 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:41.257 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:41.257 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:41.279 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:41.280 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:41.301 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:41.301 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:41.313 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:41.313+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:41.322 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:41.322 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:41.342 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:41.342 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:41.361 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:41.361 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:41.361 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:41.363 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:41.363 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:41.369 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:41.384 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:41.385 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:41.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:41.384+0000 7fa1edbac640 0 -- 172.21.15.27:0/3999602253 >> v2:172.21.15.159:6805/4073091144 conn(0x7fa1e8222200 msgr2=0x7fa1e82245f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:41.414 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:41.412+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.475 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.74089613199999, 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:52:41.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:52:41.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:52:41.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.74081873099999, 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.482 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:52:41.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.74066619300001, 2024-09-13T13:52:41.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:52:41.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:52:41.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.494 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.74056015400001, 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.495 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:52:41.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:52:41.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:41.501 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.732221076, 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.502 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:52:41.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.70485436600001, 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.506 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:52:41.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:52:41.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.70308776799999, 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:52:41.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:52:41.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:52:41.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:52:41.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:41.518 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.69820842600001, 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:41.519 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:52:41.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.68769670099999, 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:52:41.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:52:41.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.66343941100001, 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.531 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:52:41.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:52:41.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:52:41.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.66055436600001, 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.537 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.538 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:52:41.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:52:41.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:41.543 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.65835034099999, 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:41.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:52:41.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:52:41.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.548 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:52:41.549 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.651359697, 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.550 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.551 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:52:41.552 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.553 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 148.546368083, 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.554 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.555 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.556 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:52:41.557 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.558 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:52:41.559 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:41.560 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.722778222, 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:41.561 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.562 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.563 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:52:41.564 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:52:41.565 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.566 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.722745432, 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:52:41.567 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.568 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:52:41.569 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:41.570 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:52:41.571 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:41.572 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 110.606985925, 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.573 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.574 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.575 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.576 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 105.68238323600001, 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.577 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.578 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:52:41.579 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:52:41.580 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.581 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:52:41.582 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.583 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.732173489000004, 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.584 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.585 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.586 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:52:41.587 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3286 3.2 e114/73 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head v 114'133, mlcod=105'131)", 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 70.505204196999998, 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74551292800000002, 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.588 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.589 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.73e-07 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.968632+0000", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075455699999999999 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711912+0000", 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74328065200000004 2024-09-13T13:52:41.590 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711933+0000", 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1279000000000001e-05 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712080+0000", 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014671199999999999 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.591 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712101+0000", 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0339999999999998e-05 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713357+0000", 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012564550000000001 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713390+0000", 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2759999999999998e-05 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.592 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.593 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.593 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:41.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:41.595 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:41.593+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:41.608 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:41.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:41.685 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:41.683+0000 7f4ffbd7a640 -1 osd.1 173 handle_osd_map injecting map crc failure 2024-09-13T13:52:41.695 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:41.695+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:41.739 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.739 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:41.739 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:41.739 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:41.747 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:41.796 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:41.795+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:41.850 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.850 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:41.850 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:41.851 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:41.851 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:41.859 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:41.896 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:41.896+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:41.963 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:41.963 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:41.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:41.963 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:41.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.21467767199999, 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:52:41.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.21170106100001, 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:41.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:52:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:41.973 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.20972689000001, 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:52:41.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.17099636699999, 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.978 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.980 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:52:41.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.16709507600001, 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:52:41.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.987 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.04593008800001, 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.988 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:52:41.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:52:41.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.994 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.04112919299999, 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:41.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:52:41.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:41.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:41.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 147.886490356, 2024-09-13T13:52:42.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:52:42.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.004 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:52:42.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 133.70662887, 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.008 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:52:42.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:52:42.013 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 120.673537514, 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:42.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.017 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 106.164897259, 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:52:42.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 80.050869999, 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:52:42.022 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:42.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:52:42.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 79.053267640000001, 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.030 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:52:42.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4028 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.890725854000003, 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010533003000000001, 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.037 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4028 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070849+0000", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070861+0000", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1973999999999999e-05 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070881+0000", 2024-09-13T13:52:42.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070894+0000", 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2204999999999999e-05 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070965+0000", 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1613999999999996e-05 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070971+0000", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.395e-06 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077011+0000", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006040035 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077114+0000", 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103727 2024-09-13T13:52:42.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081381+0000", 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042663370000000003 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4029 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.890641809999998, 2024-09-13T13:52:42.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012297313000000001, 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4029 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:42.043 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7400000000000003e-07 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.044 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070935+0000", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1089999999999998e-06 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070941+0000", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070945+0000", 2024-09-13T13:52:42.045 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2980000000000002e-06 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071034+0000", 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8661999999999999e-05 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071041+0000", 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.046 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081442+0000", 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010401216 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081561+0000", 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119207 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083229+0000", 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001667948 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.047 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4030 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 40.890620695000003, 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722845, 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.048 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4030 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.049 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070955+0000", 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5060000000000001e-06 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.050 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070958+0000", 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3160000000000001e-06 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070963+0000", 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8500000000000002e-06 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.051 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071060+0000", 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7565999999999994e-05 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071063+0000", 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0479999999999999e-06 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.052 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083261+0000", 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012197846 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083377+0000", 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011555100000000001 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083676+0000", 2024-09-13T13:52:42.053 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029883399999999999 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4357 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.208754045999999, 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012624672, 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.054 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4357 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.055 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752820+0000", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1399999999999997e-07 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752827+0000", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5010000000000001e-06 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752834+0000", 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:52:42.056 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752838+0000", 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752943+0000", 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010495199999999999 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.057 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752947+0000", 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8789999999999996e-06 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762157+0000", 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092093769999999995 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762260+0000", 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103145 2024-09-13T13:52:42.058 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765444+0000", 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031843850000000001 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4358 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.208710401000001, 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013684981000000001, 2024-09-13T13:52:42.059 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4358 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.060 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752864+0000", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4700000000000001e-07 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752867+0000", 2024-09-13T13:52:42.061 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4350000000000001e-06 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752874+0000", 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3869999999999997e-06 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752880+0000", 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.062 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752994+0000", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752999+0000", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0270000000000003e-06 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765566+0000", 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012566623000000001 2024-09-13T13:52:42.063 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765652+0000", 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6439000000000004e-05 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.766548+0000", 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000895753 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.064 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4422 4.6 4:6ce02a65:::smithi15938319-20:head [tier-promote] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 25.041337522999999, 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722221000000001, 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.065 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4422 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.066 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920238+0000", 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.618e-06 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920244+0000", 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3929999999999996e-06 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.067 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920260+0000", 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6308e-05 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920270+0000", 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6490000000000003e-06 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920344+0000", 2024-09-13T13:52:42.068 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3979000000000004e-05 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920351+0000", 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9369999999999998e-06 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924882+0000", 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045310389999999997 2024-09-13T13:52:42.069 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924943+0000", 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0618000000000001e-05 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.932958+0000", 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080156800000000007 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.070 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4423 4.6 4:6668b762:::smithi15938319-26:head [stat,delete] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 25.041274847, 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012827221, 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4423 2024-09-13T13:52:42.071 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.072 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920302+0000", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6089999999999999e-06 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920309+0000", 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.844e-06 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.073 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920314+0000", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.259e-06 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920407+0000", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3058000000000002e-05 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920414+0000", 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4159999999999998e-06 2024-09-13T13:52:42.074 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925431+0000", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005016553 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925484+0000", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3507000000000001e-05 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925679+0000", 2024-09-13T13:52:42.075 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019484 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.928735+0000", 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030555719999999999 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933007+0000", 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042717700000000003 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.076 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933071+0000", 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3935000000000004e-05 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933126+0000", 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5358000000000001e-05 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.077 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:42.079 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:42.077+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:42.180 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:42.179+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:42.223 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.223 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:42.223 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:42.223 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.232 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:42.281 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:42.280+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:42.335 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.336 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:42.336 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:42.336 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:42.336 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.344 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:42.382 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:42.381+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:42.449 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.449 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:42.449 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:42.449 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:42.449 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.18594084399999, 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.450 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.451 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:52:42.452 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:52:42.453 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:42.454 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.184987345, 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:42.455 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.456 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:52:42.457 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.458 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 138.67415108500001, 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.459 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.460 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:52:42.461 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.462 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.463 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 115.121240512, 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.464 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:52:42.465 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.466 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:52:42.467 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.468 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 115.12049162300001, 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.469 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:52:42.470 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.471 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.472 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.473 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 111.702157441, 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.474 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:42.475 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.476 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.477 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.478 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 111.697029083, 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.479 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.480 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.481 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:52:42.482 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 111.696117527, 2024-09-13T13:52:42.483 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.484 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:52:42.485 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.486 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.487 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 100.598757095, 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.488 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:52:42.489 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.490 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:52:42.491 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.492 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 100.597989047, 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.493 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:52:42.494 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.495 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.496 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 95.241675767000004, 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.497 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.498 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.499 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.500 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:52:42.501 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 65.034268502000003, 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.502 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.503 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:52:42.504 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.505 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.506 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.141583523000001, 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.507 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:52:42.508 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.509 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.510 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.511 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.489991871999997, 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.512 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:52:42.513 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.514 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:52:42.515 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:52:42.516 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:256 4.8 e150/34 4:165e03eb:::smithi15938319-32:2c v 150'186, mlcod=148'183)", 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.690680834999998, 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045821789999999996, 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.517 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.518 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.519 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756996+0000", 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.98e-07 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757009+0000", 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2948e-05 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.520 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757049+0000", 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0793000000000001e-05 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757058+0000", 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8380000000000008e-06 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.521 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757115+0000", 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6483000000000001e-05 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757139+0000", 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4167000000000001e-05 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761524+0000", 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004384888 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.522 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761578+0000", 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3863999999999998e-05 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:257 4.8 e150/34 4:1b24d731:::smithi15938319-13:2c v 150'188, mlcod=148'183)", 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 29.689640147999999, 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063443070000000004, 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.523 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.524 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758037+0000", 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9999999999999996e-07 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758044+0000", 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.759e-06 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.525 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758116+0000", 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1913000000000001e-05 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758123+0000", 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.763e-06 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.526 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758187+0000", 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3283999999999999e-05 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758211+0000", 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4728000000000001e-05 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764325+0000", 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006114113 2024-09-13T13:52:42.527 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764380+0000", 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4947000000000001e-05 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:262 4.8 e154/34 4:1b24d731:::smithi15938319-13:26 v 154'194, mlcod=150'186)", 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 25.681698111999999, 2024-09-13T13:52:42.528 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058817069999999999, 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:42.529 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765979+0000", 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1999999999999999e-07 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765984+0000", 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.530 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766015+0000", 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1711000000000002e-05 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766021+0000", 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0009999999999998e-06 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.531 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766125+0000", 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103187 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766138+0000", 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3067999999999999e-05 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771829+0000", 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056913909999999996 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.532 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771860+0000", 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0787999999999997e-05 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:277 4.8 e163/34 4:1b24d731:::smithi15938319-13:26 v 163'204, mlcod=157'198)", 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.266484695999999, 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062937430000000001, 2024-09-13T13:52:42.533 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.534 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181200+0000", 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5780000000000001e-06 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.535 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181242+0000", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2565999999999999e-05 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181254+0000", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1151999999999999e-05 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182146+0000", 2024-09-13T13:52:42.536 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089234399999999999 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182165+0000", 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9293e-05 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187445+0000", 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005279265 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.537 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187485+0000", 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0729999999999998e-05 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4604 4.8 e163/34 4:165e03eb:::smithi15938319-32:head v 163'205, mlcod=157'198)", 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 15.265143120999999, 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004977637, 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.538 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.539 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182534+0000", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8500000000000001e-07 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182540+0000", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.959e-06 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182574+0000", 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3806999999999997e-05 2024-09-13T13:52:42.540 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182579+0000", 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.835e-06 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182627+0000", 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7926000000000002e-05 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.541 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182645+0000", 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7419e-05 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187488+0000", 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048430740000000002 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187511+0000", 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2932e-05 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.542 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4654 4.3 e168/34 4:c1ae59bf:::smithi15938319-45:head v 168'66, mlcod=163'62)", 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 10.246170958, 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047873990000000003, 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.543 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1e-07 2024-09-13T13:52:42.544 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201511+0000", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5430000000000001e-06 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201546+0000", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.447e-05 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201572+0000", 2024-09-13T13:52:42.545 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6237e-05 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201667+0000", 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5329999999999997e-05 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201688+0000", 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0568999999999999e-05 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.546 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.206244+0000", 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045564580000000002 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.206293+0000", 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8581999999999997e-05 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.547 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:42.550 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:42.548+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:42.650 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:42.648+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:42.653 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:42.653 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:42.653 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.662 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:42.751 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:42.750+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:42.806 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:42.806 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:42.806 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:42.814 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:42.852 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:42.850+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:42.919 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:4 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'2, mlcod=0'0)", 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 174.37605160800001, 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.63017036299999996, 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.920 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542014+0000", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542015+0000", 2024-09-13T13:52:42.921 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9699999999999995e-07 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:48.542021+0000", 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.144e-06 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171202+0000", 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62918043000000001 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.922 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171217+0000", 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5383000000000002e-05 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171379+0000", 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016194700000000001 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.923 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.171422+0000", 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2826e-05 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172134+0000", 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00071221399999999997 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:49.172185+0000", 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0621999999999997e-05 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.924 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:93 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'67, mlcod=25'61)", 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 172.71660954199999, 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071700469999999997, 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.925 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.927 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201456+0000", 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.928 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201457+0000", 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2599999999999999e-07 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201611+0000", 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015475100000000001 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.929 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201630+0000", 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9020000000000001e-05 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.201634+0000", 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0960000000000003e-06 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205302+0000", 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0036674950000000002 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.930 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205315+0000", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.341e-05 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208592+0000", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032769729999999999 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.208627+0000", 2024-09-13T13:52:42.931 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4175999999999999e-05 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4217.0:94 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'68, mlcod=25'61)", 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 172.715221318, 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061933800000000001, 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.932 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.933 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202845+0000", 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202846+0000", 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4e-07 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202852+0000", 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4350000000000002e-06 2024-09-13T13:52:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202884+0000", 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2344000000000001e-05 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.202891+0000", 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4620000000000001e-06 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205734+0000", 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028435980000000001 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.205747+0000", 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2408e-05 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209008+0000", 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003260771 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:49:50.209038+0000", 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0521999999999997e-05 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:271 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 150.001236019, 2024-09-13T13:52:42.937 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014543471000000001, 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 271 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.938 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916830+0000", 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916832+0000", 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.562e-06 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.939 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916837+0000", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3680000000000001e-06 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916848+0000", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1511999999999999e-05 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916870+0000", 2024-09-13T13:52:42.940 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1494000000000001e-05 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.916934+0000", 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3819999999999995e-05 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.917018+0000", 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4448000000000002e-05 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.941 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.929845+0000", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012826994 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930287+0000", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044215200000000002 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931304+0000", 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001016943 2024-09-13T13:52:42.942 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931348+0000", 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3739000000000003e-05 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931373+0000", 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5439000000000002e-05 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.943 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:321 4.9 4:977d569f:::smithi15938319-22:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 149.14739727099999, 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010755851, 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 321 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.944 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770669+0000", 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.945 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770671+0000", 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1179999999999999e-06 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770678+0000", 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3860000000000001e-06 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770694+0000", 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5886e-05 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.946 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770705+0000", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0922e-05 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770769+0000", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3948000000000005e-05 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.770835+0000", 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5649999999999997e-05 2024-09-13T13:52:42.947 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.777908+0000", 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070735329999999999 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.778875+0000", 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00096685499999999997 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.948 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781203+0000", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023276070000000002 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781317+0000", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114391 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.781425+0000", 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107555 2024-09-13T13:52:42.949 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:813 4.9 4:977d569f:::smithi15938319-22:head [read 0~2840350,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 144.727962402, 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0076418909999999996, 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.950 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 813 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.951 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190104+0000", 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190109+0000", 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7910000000000004e-06 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.952 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190119+0000", 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8490000000000001e-06 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190123+0000", 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6349999999999997e-06 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190242+0000", 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011864199999999999 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.953 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190246+0000", 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.563e-06 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195502+0000", 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052564939999999996 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.954 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.195554+0000", 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2198999999999999e-05 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197745+0000", 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0021911270000000002 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.955 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:814 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 1656210~1077644] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 144.72791445199999, 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0078523050000000004, 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 814 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.956 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190152+0000", 2024-09-13T13:52:42.957 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4999999999999998e-07 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190154+0000", 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.356e-06 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190159+0000", 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4340000000000002e-06 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.958 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190163+0000", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6890000000000003e-06 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190267+0000", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010341899999999999 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.190270+0000", 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0639999999999998e-06 2024-09-13T13:52:42.959 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197770+0000", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0074997040000000003 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.197822+0000", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2422000000000001e-05 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:18.198004+0000", 2024-09-13T13:52:42.960 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000181767 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:910 4.9 4:977d569f:::smithi15938319-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 143.039451525, 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0096678049999999998, 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.961 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 910 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:42.962 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878614+0000", 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878617+0000", 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1280000000000002e-06 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.963 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878627+0000", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0404999999999999e-05 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878645+0000", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7974000000000001e-05 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878654+0000", 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3640000000000005e-06 2024-09-13T13:52:42.964 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878707+0000", 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2227e-05 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.878771+0000", 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4678999999999995e-05 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:42.965 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887294+0000", 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0085226469999999995 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.887898+0000", 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00060379400000000005 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888217+0000", 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031976200000000002 2024-09-13T13:52:42.966 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888251+0000", 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3606000000000003e-05 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:19.888282+0000", 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1219e-05 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.967 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1992 4.9 4:977d569f:::smithi15938319-22:head [write 571471~640157] snapc d=[d,b,9,8] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.343844398, 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059992420000000001, 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.968 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1992 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574222+0000", 2024-09-13T13:52:42.969 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574224+0000", 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4760000000000001e-06 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574576+0000", 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00035188899999999998 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.970 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574595+0000", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8756000000000001e-05 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574602+0000", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5519999999999999e-06 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574671+0000", 2024-09-13T13:52:42.971 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8308999999999999e-05 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574754+0000", 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3837000000000004e-05 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.574909+0000", 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015433999999999999 2024-09-13T13:52:42.972 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.576225+0000", 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0013163459999999999 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580116+0000", 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038911010000000001 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.973 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580172+0000", 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5695000000000001e-05 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:45.580221+0000", 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940999999999998e-05 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.974 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2041 4.9 4:977d569f:::smithi15938319-22:head [tier-promote] snapc e=[e,d,b,8] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 115.61614212400001, 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0067017520000000001, 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2041 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.975 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301924+0000", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301926+0000", 2024-09-13T13:52:42.976 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1670000000000002e-06 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301934+0000", 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.2940000000000006e-06 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301953+0000", 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.8899e-05 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.977 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.301966+0000", 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2792999999999999e-05 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302041+0000", 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4672999999999995e-05 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.302048+0000", 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4139999999999997e-06 2024-09-13T13:52:42.978 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307261+0000", 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052130750000000002 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.307324+0000", 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2885999999999994e-05 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.979 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.308626+0000", 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001301551 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2057 4.9 4:977d569f:::smithi15938319-22:head [read 0~2571471,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 115.58996698199999, 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061580649999999999, 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.980 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2057 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.981 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328099+0000", 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328100+0000", 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3500000000000005e-07 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328106+0000", 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8200000000000002e-06 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.982 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328116+0000", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0669999999999999e-05 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328124+0000", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9019999999999999e-06 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333040+0000", 2024-09-13T13:52:42.983 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049159269999999996 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.333139+0000", 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8430000000000005e-05 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334257+0000", 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011183810000000001 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.984 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2058 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1169372] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 115.589925966, 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0065083880000000004, 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.985 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2058 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.986 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328140+0000", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5400000000000002e-07 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328143+0000", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8339999999999999e-06 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328149+0000", 2024-09-13T13:52:42.987 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2190000000000001e-06 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.328154+0000", 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6700000000000002e-06 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334293+0000", 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061386879999999998 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.988 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334390+0000", 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6737000000000006e-05 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.334648+0000", 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000258886 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.989 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2870 4.9 4:977d569f:::smithi15938319-22:12 [set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 90.770789058000005, 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0088359319999999995, 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2870 2024-09-13T13:52:42.990 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147277+0000", 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.991 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147280+0000", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.847e-06 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147296+0000", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6402999999999999e-05 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147332+0000", 2024-09-13T13:52:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5726000000000003e-05 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147354+0000", 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2350000000000001e-05 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147472+0000", 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118084 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.147559+0000", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7003999999999996e-05 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.154818+0000", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0072584399999999997 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.155877+0000", 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010593340000000001 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.994 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156009+0000", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013197799999999999 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156065+0000", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5899999999999997e-05 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:12.156113+0000", 2024-09-13T13:52:42.995 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7865999999999999e-05 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3275 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.975809964999996, 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0061117630000000001, 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:42.996 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3275 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.997 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942256+0000", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942258+0000", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9369999999999998e-06 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942265+0000", 2024-09-13T13:52:42.998 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0829999999999998e-06 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942282+0000", 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6880000000000001e-05 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942292+0000", 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8689999999999999e-06 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:42.999 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7336000000000005e-05 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942388+0000", 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3090000000000001e-06 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.000 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945082+0000", 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0026939160000000002 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945163+0000", 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.0728000000000004e-05 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.945384+0000", 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000221375 2024-09-13T13:52:43.001 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948177+0000", 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0027923119999999999 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948231+0000", 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3845999999999999e-05 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:43.002 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948299+0000", 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8300000000000007e-05 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948368+0000", 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8872000000000001e-05 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.003 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3276 4.9 4:977d569f:::smithi15938319-22:head [rollback 12] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.97574548, 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082025069999999995, 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3276 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.004 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.005 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942321+0000", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7100000000000001e-07 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942325+0000", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.512e-06 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942335+0000", 2024-09-13T13:52:43.006 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0016e-05 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942341+0000", 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4910000000000002e-06 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942471+0000", 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013015300000000001 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.007 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942478+0000", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.55e-06 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946541+0000", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040631779999999998 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946640+0000", 2024-09-13T13:52:43.008 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8694000000000001e-05 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.946923+0000", 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028351899999999998 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948388+0000", 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001464414 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.009 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950419+0000", 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0020313689999999999 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950465+0000", 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5519999999999998e-05 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.010 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.950523+0000", 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.842e-05 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3277 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 24=[24,23,22,20,1e,1c,1b,1a,17,14,12] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 72.975687159000003, 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010801006, 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.011 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3277 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.012 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942379+0000", 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942380+0000", 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2200000000000003e-07 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.013 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942384+0000", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1949999999999996e-06 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942391+0000", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0339999999999999e-06 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942396+0000", 2024-09-13T13:52:43.014 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8409999999999999e-06 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942508+0000", 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000112458 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.942512+0000", 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.579e-06 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.015 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947640+0000", 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0051287859999999998 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947737+0000", 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6329999999999994e-05 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.016 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.947934+0000", 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019687900000000001 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.948478+0000", 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000544448 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:43.017 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953066+0000", 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00458831 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953129+0000", 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2280999999999995e-05 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:29.953180+0000", 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:43.018 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3520 4.9 4:977d569f:::smithi15938319-22:head [read 627200~1132544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 61.943484028, 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062848280000000001, 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:43.019 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3520 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.020 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974582+0000", 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974583+0000", 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.077e-06 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974589+0000", 2024-09-13T13:52:43.021 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4879999999999998e-06 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974600+0000", 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1232e-05 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974609+0000", 2024-09-13T13:52:43.022 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3430000000000002e-06 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974667+0000", 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7388e-05 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.974672+0000", 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6930000000000004e-06 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.023 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979695+0000", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0050231449999999997 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.979758+0000", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.2232999999999999e-05 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.980867+0000", 2024-09-13T13:52:43.024 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001109229 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3660 4.9 4:977d569f:::smithi15938319-22:head [append 0~0] snapc 2c=[2c,2b,2a,29,28,27,26,25,24,23,22,1e,1a,14,12] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 55.214741721999999, 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068015810000000001, 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.025 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3660 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.026 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703324+0000", 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703325+0000", 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.91e-07 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703326+0000", 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6759999999999999e-06 2024-09-13T13:52:43.027 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703331+0000", 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6879999999999998e-06 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.703335+0000", 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8940000000000003e-06 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.028 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706175+0000", 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028405430000000001 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706229+0000", 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3813000000000001e-05 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706343+0000", 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000114123 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.029 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.706835+0000", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00049211600000000002 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710024+0000", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003188116 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710080+0000", 2024-09-13T13:52:43.030 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6168999999999997e-05 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:47.710126+0000", 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6152000000000001e-05 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4099 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1749291] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:43.031 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.837634104999999, 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059876800000000004, 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4099 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:43.032 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080432+0000", 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8599999999999998e-07 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.033 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080436+0000", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.309e-06 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080443+0000", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6920000000000002e-06 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080449+0000", 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6980000000000003e-06 2024-09-13T13:52:43.034 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080533+0000", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3633000000000004e-05 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080540+0000", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7830000000000001e-06 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086023+0000", 2024-09-13T13:52:43.035 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054832650000000002 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086131+0000", 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000107798 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086420+0000", 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00028891600000000001 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.036 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4100 4.9 4:977d569f:::smithi15938319-22:head [sparse-read 0~1749291] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e140)", 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 39.837606710999999, 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0066065960000000002, 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:43.037 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4100 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080459+0000", 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.038 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080460+0000", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6199999999999998e-07 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080462+0000", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1560000000000002e-06 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080468+0000", 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6380000000000001e-06 2024-09-13T13:52:43.039 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080474+0000", 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8920000000000003e-06 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080576+0000", 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000101439 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:43.040 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.080582+0000", 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5769999999999999e-06 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086462+0000", 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058799810000000003 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.086579+0000", 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000116864 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:43.041 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:03.087066+0000", 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048658700000000002 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:43.042 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:43.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:43.044 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:43.043+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:43.144 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:43.143+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:43.164 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:43.164 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:43.164 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:43.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:43.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:52:43.245 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:43.245+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:43.322 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:43.323 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:43.323 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:43.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:43.323 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:43.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:43.377 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:43.376+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:43.448 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:43.446+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:43.455 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 20, 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 600, 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [ 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1073 4.5 e48/47 4:aebdca62:::smithi15938319-23:head v 48'32, mlcod=48'29)", 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 139.100883469, 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034204219999999998, 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.456 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353414+0000", 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.457 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353415+0000", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353424+0000", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.1120000000000002e-06 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353454+0000", 2024-09-13T13:52:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0057e-05 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353468+0000", 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.397e-05 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353628+0000", 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00016041100000000001 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.459 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.353654+0000", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5854999999999999e-05 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356795+0000", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0031411579999999998 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.356834+0000", 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9057999999999998e-05 2024-09-13T13:52:43.460 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:60 4.5 e48/47 4:a1652551:::smithi15938319-33:head v 48'33, mlcod=48'31)", 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 138.461431288, 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.007660816, 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.461 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992866+0000", 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.462 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.992867+0000", 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.333e-06 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995035+0000", 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0021676579999999998 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995073+0000", 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.7509000000000003e-05 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.463 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995083+0000", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0501e-05 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995136+0000", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2685999999999997e-05 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995154+0000", 2024-09-13T13:52:43.464 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8363e-05 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000485+0000", 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0053304779999999996 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.465 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000527+0000", 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.2287999999999999e-05 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "pg_update_log_missing(4.5 epoch 48/47 rep_tid 61 entries 48'34 (0'0) error 4:a1652551:::smithi15938319-33:head by client.4571.0:1149 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 48'31)", 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 138.45883433099999, 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0052481710000000003, 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.466 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995463+0000", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.467 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995464+0000", 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995471+0000", 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1940000000000001e-06 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995494+0000", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.287e-05 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.995502+0000", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.1499999999999999e-06 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:24.997361+0000", 2024-09-13T13:52:43.469 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001859362 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:25.000711+0000", 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0033500840000000001 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:43.470 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 134.719124588, 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029737629999999999, 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.471 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735173+0000", 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4200000000000001e-07 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.472 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.473 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735180+0000", 2024-09-13T13:52:43.473 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1439999999999997e-06 2024-09-13T13:52:43.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.473 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735198+0000", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7966e-05 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.735217+0000", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9151999999999999e-05 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.736839+0000", 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016218879999999999 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.474 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.738147+0000", 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001307171 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPull(4.5 e53/52 cost 1049576)", 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 134.705679863, 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0029417319999999999, 2024-09-13T13:52:43.475 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "reached pg", 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748617+0000", 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748618+0000", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748633+0000", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4718e-05 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748651+0000", 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7978999999999998e-05 2024-09-13T13:52:43.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.748656+0000", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9459999999999997e-06 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.750265+0000", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016092089999999999 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:28.751559+0000", 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.001294384 2024-09-13T13:52:43.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:7, version: 60'59, data_included: [221184~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:7@60'59, size: 2537824, copy_subset: [(0, 2537824)], clone_subset: {}, snapset: 7=[]:{7: [7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1269760, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 121.81073315, 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032730889999999999, 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.479 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643564+0000", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.643565+0000", 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:43.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646155+0000", 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025901470000000001 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646188+0000", 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3399000000000003e-05 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.481 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646204+0000", 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6112000000000001e-05 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646288+0000", 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.3219000000000004e-05 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646306+0000", 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.8181000000000001e-05 2024-09-13T13:52:43.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.646837+0000", 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000531482 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "MOSDPGPush(4.5 64/63 [PushOp(4:a1652551:::smithi15938319-33:head, version: 60'60, data_included: [1962249~409893,3117056~580451], data_size: 990344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a1652551:::smithi15938319-33:head@60'60, size: 3697507, copy_subset: [(413472, 796773), (1710446, 661696), (2537824, 1159683)], clone_subset: {4:a1652551:::smithi15938319-33:7: [(0, 413472), (1210245, 500201), (2372142, 165682)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3697507, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962249, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 121.76701806600001, 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00296737, 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "started", 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.484 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687279+0000", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.687280+0000", 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.5300000000000004e-07 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688170+0000", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00089030999999999995 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688201+0000", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0947e-05 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688213+0000", 2024-09-13T13:52:43.486 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2166e-05 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.7826999999999998e-05 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.688315+0000", 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.3890000000000001e-05 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.487 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.690247+0000", 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0019316769999999999 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:1844 4.5 e64/63 4:aae6a8d6:::smithi15938319-43:head v 64'68, mlcod=0'0)", 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 121.573204483, 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.003721785, 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.488 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881093+0000", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881094+0000", 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.6799999999999999e-07 2024-09-13T13:52:43.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881101+0000", 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.9619999999999996e-06 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881130+0000", 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9094999999999998e-05 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.490 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881144+0000", 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.4518999999999999e-05 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881190+0000", 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.5549e-05 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.881216+0000", 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6239000000000001e-05 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.491 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884773+0000", 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035567099999999998 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:41.884815+0000", 2024-09-13T13:52:43.492 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1842999999999999e-05 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2171 4.5 e75/63 4:a9f43e9a:::smithi15938319-36:head v 75'96, mlcod=70'92)", 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 111.69281264, 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034223410000000002, 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.493 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.494 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.761485+0000", 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1399999999999998e-07 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762677+0000", 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0011921810000000001 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762703+0000", 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:43.495 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762710+0000", 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.1579999999999999e-06 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762778+0000", 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.496 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.762805+0000", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7832000000000001e-05 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764874+0000", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0020681710000000002 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:51.764907+0000", 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.341e-05 2024-09-13T13:52:43.497 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2285 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'111, mlcod=78'107)", 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.47772129499999, 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0041178459999999997, 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.498 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976576+0000", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976577+0000", 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.3200000000000005e-07 2024-09-13T13:52:43.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976582+0000", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4809999999999997e-06 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976615+0000", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3104e-05 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976622+0000", 2024-09-13T13:52:43.500 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6139999999999997e-06 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976670+0000", 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8151e-05 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.976689+0000", 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.9259e-05 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980651+0000", 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0039612229999999998 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980694+0000", 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3381999999999999e-05 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2286 4.5 e78/63 4:aae6a8d6:::smithi15938319-43:head v 78'112, mlcod=78'107)", 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.476842866, 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035509550000000002, 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:43.503 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977454+0000", 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977455+0000", 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.4099999999999999e-07 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.504 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977459+0000", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.4610000000000001e-06 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977483+0000", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.3295e-05 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977489+0000", 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7459999999999997e-06 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977546+0000", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.7334999999999999e-05 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.977558+0000", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.2113000000000001e-05 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.980975+0000", 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034168839999999998 2024-09-13T13:52:43.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:54.981005+0000", 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.0580000000000002e-05 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2295 4.5 e78/63 4:a9f43e9a:::smithi15938319-36:head v 78'114, mlcod=78'111)", 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 108.453127368, 2024-09-13T13:52:43.507 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0066501279999999999, 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.508 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001170+0000", 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.001171+0000", 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.2600000000000001e-07 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002210+0000", 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0010393869999999999 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002238+0000", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.7843000000000001e-05 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002249+0000", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.064e-05 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002326+0000", 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.7033999999999995e-05 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.002343+0000", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7067000000000001e-05 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007784+0000", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0054414320000000004 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:55.007820+0000", 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.5899000000000003e-05 2024-09-13T13:52:43.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:143 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:c v 80'121, mlcod=79'118)", 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.679765319, 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0046598719999999998, 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.512 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.513 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774532+0000", 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.1699999999999999e-07 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.774539+0000", 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.5740000000000001e-06 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775178+0000", 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00063906 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.514 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775199+0000", 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1009000000000002e-05 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775319+0000", 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000119886 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.515 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775343+0000", 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4226000000000001e-05 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779145+0000", 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0038022640000000001 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.516 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.779192+0000", 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.6436000000000001e-05 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:144 4.5 e80/63 4:a1652551:::smithi15938319-33:e v 80'123, mlcod=79'118)", 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.67916443, 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0065740950000000003, 2024-09-13T13:52:43.517 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775133+0000", 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.518 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775134+0000", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.8000000000000004e-07 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775141+0000", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 7.2259999999999999e-06 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775261+0000", 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000120213 2024-09-13T13:52:43.519 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775267+0000", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.0360000000000003e-06 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775852+0000", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00058448599999999995 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.775872+0000", 2024-09-13T13:52:43.520 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0469e-05 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781664+0000", 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0057913060000000004 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.781707+0000", 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.3479000000000002e-05 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.521 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:145 4.5 e80/63 4:a9f43e9a:::smithi15938319-36:11 v 80'125, mlcod=80'121)", 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.672011853, 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0034284490000000001, 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.522 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782285+0000", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782287+0000", 2024-09-13T13:52:43.523 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.1650000000000001e-06 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782296+0000", 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.918e-06 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782336+0000", 2024-09-13T13:52:43.524 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0321000000000002e-05 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782345+0000", 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782431+0000", 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.5958000000000006e-05 2024-09-13T13:52:43.525 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.782454+0000", 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2594000000000001e-05 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.526 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785675+0000", 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0032206079999999998 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:56.785714+0000", 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.9297e-05 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.527 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2404 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'127, mlcod=80'123)", 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.36624743900001, 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0027014560000000001, 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.528 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.529 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088050+0000", 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088051+0000", 2024-09-13T13:52:43.530 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088933+0000", 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00088239000000000004 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.531 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088964+0000", 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.1021e-05 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.088973+0000", 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4719999999999995e-06 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.532 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089025+0000", 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.2407999999999997e-05 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.089046+0000", 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.0792999999999999e-05 2024-09-13T13:52:43.533 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090722+0000", 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0016761759999999999 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.534 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.090751+0000", 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.9175000000000001e-05 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(client.4571.0:2405 4.5 e80/63 4:aae6a8d6:::smithi15938319-43:head v 80'128, mlcod=80'125)", 2024-09-13T13:52:43.535 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 106.358615907, 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0025095180000000001, 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.536 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095681+0000", 2024-09-13T13:52:43.537 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.095686+0000", 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8450000000000002e-06 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096368+0000", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00068220800000000001 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096391+0000", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2665999999999999e-05 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096396+0000", 2024-09-13T13:52:43.541 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0329999999999999e-06 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096455+0000", 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.8480999999999998e-05 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.096479+0000", 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.4091e-05 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098165+0000", 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.00168651 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:50:57.098191+0000", 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.5684e-05 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.543 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:158 4.5 e84/63 4:aae6a8d6:::smithi15938319-43:c v 84'141, mlcod=81'138)", 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 102.647144764, 2024-09-13T13:52:43.544 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.004621539, 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.545 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807153+0000", 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1299999999999999e-07 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.546 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807157+0000", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.8349999999999997e-06 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807193+0000", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.6859000000000001e-05 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807204+0000", 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.0774000000000001e-05 2024-09-13T13:52:43.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807253+0000", 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.8900000000000003e-05 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.807270+0000", 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.7034e-05 2024-09-13T13:52:43.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811727+0000", 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0044564000000000001 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.811774+0000", 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.7524000000000001e-05 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:159 4.5 e84/63 4:aebdca62:::smithi15938319-23:9 v 84'143, mlcod=81'138)", 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 102.646209797, 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0086742480000000007, 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.550 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808088+0000", 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.8400000000000004e-07 2024-09-13T13:52:43.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808105+0000", 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 1.6773999999999999e-05 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808138+0000", 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.3330000000000001e-05 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808147+0000", 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 9.0499999999999997e-06 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808331+0000", 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.000183358 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.553 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.808352+0000", 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.1237999999999999e-05 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816713+0000", 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0083605720000000001 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.816762+0000", 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.9242000000000001e-05 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "description": "osd_repop(osd.10.0:170 4.5 e84/63 4:a9f43e9a:::smithi15938319-36:a v 84'147, mlcod=84'141)", 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "initiated_at": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "age": 102.635526963, 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0037313229999999999, 2024-09-13T13:52:43.555 INFO:teuthology.orchestra.run.smithi027.stdout: "continuous": false, 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "type_data": { 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "events": [ 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "initiated", 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "header_read", 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818770+0000", 2024-09-13T13:52:43.556 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "throttled", 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818771+0000", 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4099999999999997e-07 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "all_read", 2024-09-13T13:52:43.557 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818778+0000", 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 6.6610000000000003e-06 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "dispatched", 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818812+0000", 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 3.4217999999999997e-05 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "queued_for_pg", 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818821+0000", 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 8.4100000000000008e-06 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "reached_pg", 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818871+0000", 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 5.0574000000000001e-05 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "started", 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.818893+0000", 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 2.2147e-05 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.559 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "commit_sent", 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822461+0000", 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 0.0035682230000000001 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "event": "done", 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "time": "2024-09-13T13:51:00.822502+0000", 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: "duration": 4.0249e-05 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-13T13:52:43.561 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-13T13:52:43.561 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:43.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:43.563 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:43.561+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:43.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:43.663 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:43.662+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:43.711 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:43.711 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:43.711 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:52:43.711 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:43.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:43.752 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:43.754 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:43.752+0000 7f26bcfe2640 -1 osd.3 174 handle_osd_map injecting map crc failure 2024-09-13T13:52:43.763 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:43.763+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:43.787 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:43.787 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:43.810 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:43.810 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:43.830 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:52:43.830 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:52:43.830 INFO:teuthology.orchestra.run.smithi027.stdout: "complaint_time": 30, 2024-09-13T13:52:43.830 INFO:teuthology.orchestra.run.smithi027.stdout: "num_blocked_ops": 0 2024-09-13T13:52:43.831 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:52:43.831 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:43.840 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:43.854 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:43.855 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:43.855 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:43.864 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:43.864+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:43.877 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:43.878 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:43.899 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:43.900 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:43.921 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:43.922 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:43.944 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:43.944 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:43.963 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:43.963 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:43.964 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:43.963+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:43.965 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:43.965 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:43.972 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:43.987 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:44.010 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:44.010 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:44.031 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:44.031 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:44.053 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:44.053 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:44.065 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:44.064+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:44.074 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:44.075 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:44.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:44.078+0000 7f18d1d9b640 0 -- 172.21.15.27:0/2271563777 >> v2:172.21.15.159:6805/4073091144 conn(0x7f18d4222610 msgr2=0x7f18d4224a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:44.095 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.095 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.101 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:44.101 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:44.104 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:44.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:44.167 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:44.165+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:44.251 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.251 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:44.267 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:44.266+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:44.366 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.367 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.368 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:44.367+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:44.375 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:44.468 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:44.467+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:44.490 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.490 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:44.569 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:44.569+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:44.602 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.602 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:44.670 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:44.669+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:44.714 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.714 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:44.772 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:44.770+0000 7f1a5fcf1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33151) UID: 0 2024-09-13T13:52:44.825 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.825 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:44.872 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:44.871+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:44.927 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:44.926+0000 7fe0f9e03640 -1 osd.2 174 handle_osd_map injecting map crc failure 2024-09-13T13:52:44.927 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:44.925+0000 7fe857f4f640 -1 osd.7 174 handle_osd_map injecting map crc failure 2024-09-13T13:52:44.973 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:44.972+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:44.977 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:44.977 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:44.986 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:45.074 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:45.073+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:45.089 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.089 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.098 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:45.176 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:45.175+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:45.241 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.242 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:45.276 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:45.275+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:45.353 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.353 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:45.377 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:45.376+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:45.463 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.464 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:45.478 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:45.477+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:45.575 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.575 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.580 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:52:45.578+0000 7f3c40f33640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33147) UID: 0 2024-09-13T13:52:45.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:45.680 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:45.678+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:45.726 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.726 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:45.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T13:52:45.813 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:45.812+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:45.838 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.839 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:45.881 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:45.880+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:45.970 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:45.971 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:45.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:45.982 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:45.981+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:46.083 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:46.082+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:46.088 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:46.088 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:46.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:46.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T13:52:46.184 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:46.182+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:46.240 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:46.240 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:46.241 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is set 2024-09-13T13:52:46.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:46.265 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:46.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:46.286 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:46.287 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:46.285+0000 7fe0f9e03640 -1 osd.2 175 handle_osd_map injecting map crc failure 2024-09-13T13:52:46.287 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:46.286+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:46.308 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:46.308 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:46.308 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:46.329 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:46.329 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:46.352 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:46.352 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:46.354 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:46.354 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:46.355 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:46.355 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:46.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:46.378 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:46.385 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:46.384+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:46.398 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:46.398 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:46.419 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:46.440 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:46.440 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:46.461 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:46.482 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:46.482 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:46.486 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:46.485+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:46.503 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:46.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:46.502+0000 7f129b7fe640 0 -- 172.21.15.27:0/2924341938 >> v2:172.21.15.122:6808/1255553857 conn(0x7f129c21cc60 msgr2=0x7f129c21f070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:46.507 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:46.507 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:46.507 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:46.507 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:46.507 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:46.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:46.601 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:46.600+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:46.661 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:46.661 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:46.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:46.661 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:46.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.662 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:4 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=8b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T13:52:46.662 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:46.662 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 178.11667469599999, 2024-09-13T13:52:46.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63113447199999995, 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:46.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 4 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541670+0000", 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541671+0000", 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.353e-06 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541677+0000", 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541691+0000", 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4025999999999999e-05 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541697+0000", 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5239999999999997e-06 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541737+0000", 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9895999999999999e-05 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541791+0000", 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3888000000000003e-05 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:46.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.541946+0000", 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502499999999999 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:48.542472+0000", 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052562799999999999 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172632+0000", 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63016074899999996 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172718+0000", 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5265000000000004e-05 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:49.172804+0000", 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6525000000000003e-05 2024-09-13T13:52:46.669 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4217.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 176.06234880700001, 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.68564075099999999, 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.670 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4217", 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/422396953", 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 124 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.595996+0000", 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-09-13T13:52:46.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596002+0000", 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9499999999999998e-06 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596010+0000", 2024-09-13T13:52:46.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3839999999999999e-06 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.596014+0000", 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0899999999999998e-06 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605724+0000", 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097102939999999995 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605781+0000", 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7203000000000001e-05 2024-09-13T13:52:46.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 15", 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.605876+0000", 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4171000000000004e-05 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.606503+0000", 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062716300000000003 2024-09-13T13:52:46.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281518+0000", 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67501575300000005 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281579+0000", 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0773999999999998e-05 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281636+0000", 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7185999999999998e-05 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4217.0:124 1.0 e26/23 ondisk, result = 0)", 2024-09-13T13:52:46.678 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:46.679 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 176.04897041699999, 2024-09-13T13:52:46.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67226954100000003, 2024-09-13T13:52:46.679 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.679 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.679 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609374+0000", 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609375+0000", 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9800000000000003e-07 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.609380+0000", 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7940000000000002e-06 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281110+0000", 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.67173032700000002 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281119+0000", 2024-09-13T13:52:46.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9719999999999998e-06 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281482+0000", 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036362100000000001 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281512+0000", 2024-09-13T13:52:46.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.938e-05 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:51.281644+0000", 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013174899999999999 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:832 4.4 4:21c00763:::smithi15938319-21:head [tier-promote] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 148.23563748000001, 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63045760500000003, 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 832 2024-09-13T13:52:46.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422707+0000", 2024-09-13T13:52:46.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422708+0000", 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2300000000000001e-07 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422710+0000", 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.638e-06 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422722+0000", 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2279000000000001e-05 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.422731+0000", 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0730000000000005e-06 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424414+0000", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001683765 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.424511+0000", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6903000000000004e-05 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:19.053165+0000", 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.62865342400000002 2024-09-13T13:52:46.690 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1227 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43022547300001, 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50342587699999997, 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.691 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1227 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228119+0000", 2024-09-13T13:52:46.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228120+0000", 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228128+0000", 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2830000000000001e-06 2024-09-13T13:52:46.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228138+0000", 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9089999999999995e-06 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228144+0000", 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9859999999999999e-06 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228180+0000", 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6031000000000001e-05 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228187+0000", 2024-09-13T13:52:46.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5060000000000001e-06 2024-09-13T13:52:46.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728528+0000", 2024-09-13T13:52:46.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50034159899999997 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728665+0000", 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013653600000000001 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731414+0000", 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027490000000000001 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731545+0000", 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013091599999999999 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.698 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1228 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43018830099999, 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50352607699999996, 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.699 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1228 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228156+0000", 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228157+0000", 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-09-13T13:52:46.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228159+0000", 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3659999999999999e-06 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228164+0000", 2024-09-13T13:52:46.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.006e-06 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228169+0000", 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9579999999999998e-06 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228209+0000", 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0120000000000002e-05 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228216+0000", 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8249999999999999e-06 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728858+0000", 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50064189400000003 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728903+0000", 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4715999999999997e-05 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731594+0000", 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026913060000000001 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731682+0000", 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8391000000000001e-05 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1229 4.2 4:4eae1ead:::smithi15938319-37:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43016757300001, 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50367317899999997, 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.707 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1229 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228177+0000", 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228181+0000", 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5379999999999998e-06 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228188+0000", 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8619999999999997e-06 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228193+0000", 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1039999999999998e-06 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228230+0000", 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6860999999999998e-05 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228233+0000", 2024-09-13T13:52:46.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.698e-06 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728934+0000", 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50070099899999998 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728961+0000", 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6511000000000001e-05 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731746+0000", 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002785193 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.731850+0000", 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104041 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.713 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1241 4.5 4:a1652551:::smithi15938319-33:head [read 1353650~1184174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43007127199999, 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53506652099999996, 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1241 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228273+0000", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228274+0000", 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-09-13T13:52:46.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228277+0000", 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1219999999999999e-06 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228283+0000", 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3910000000000003e-06 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228287+0000", 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9469999999999996e-06 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228335+0000", 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7432999999999998e-05 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228342+0000", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734373+0000", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50603089800000001 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734499+0000", 2024-09-13T13:52:46.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126401 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760408+0000", 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025908944999999999 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760538+0000", 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007700000000001 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763340+0000", 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002801446 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1242 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 827996~1691044] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43004862800001, 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53923328800000003, 2024-09-13T13:52:46.720 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1242 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228296+0000", 2024-09-13T13:52:46.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7899999999999999e-07 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228298+0000", 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6950000000000001e-06 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228305+0000", 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5349999999999999e-06 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228311+0000", 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3689999999999998e-06 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228372+0000", 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1329999999999997e-05 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228378+0000", 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3789999999999997e-06 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734740+0000", 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.50636285299999995 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734788+0000", 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7856999999999999e-05 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763381+0000", 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028592649000000001 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.763499+0000", 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011848199999999999 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.767529+0000", 2024-09-13T13:52:46.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040297600000000003 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1243 4.5 4:a1652551:::smithi15938319-33:head [sparse-read 1292750~1226290] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.43002259799999, 2024-09-13T13:52:46.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54025780499999998, 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.729 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1243 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228322+0000", 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9499999999999998e-07 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228324+0000", 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.153e-06 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228331+0000", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228336+0000", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228395+0000", 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9311999999999997e-05 2024-09-13T13:52:46.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.228401+0000", 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9190000000000004e-06 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734838+0000", 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.506436687 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.734885+0000", 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6554000000000002e-05 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768340+0000", 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033455099000000002 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768396+0000", 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5887999999999998e-05 2024-09-13T13:52:46.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.768580+0000", 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018416799999999999 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.735 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1265 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.29052173100001, 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38759265799999998, 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.736 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1265 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367823+0000", 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367826+0000", 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9739999999999998e-06 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367840+0000", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4276e-05 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367900+0000", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9951999999999998e-05 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367909+0000", 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0350000000000007e-06 2024-09-13T13:52:46.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3973000000000006e-05 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367983+0000", 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5799999999999998e-06 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728989+0000", 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.361006877 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729095+0000", 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105191 2024-09-13T13:52:46.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748246+0000", 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019151400999999998 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748365+0000", 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119047 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755415+0000", 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070503520000000002 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1266 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 2756562~621612,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.29041478400001, 2024-09-13T13:52:46.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38937840699999998, 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1266 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:46.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367930+0000", 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1099999999999996e-07 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367933+0000", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0790000000000001e-06 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367940+0000", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5180000000000001e-06 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367945+0000", 2024-09-13T13:52:46.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3530000000000002e-06 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368026+0000", 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0951999999999993e-05 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368036+0000", 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9790000000000005e-06 2024-09-13T13:52:46.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729128+0000", 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36109140499999998 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729152+0000", 2024-09-13T13:52:46.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4102000000000001e-05 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755470+0000", 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026318764000000001 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.755612+0000", 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141698 2024-09-13T13:52:46.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757308+0000", 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016960460000000001 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1267 4.2 4:4dc7b2b1:::smithi15938319-14:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.290385525, 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38929330699999998, 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.751 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1267 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367959+0000", 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5699999999999998e-07 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367962+0000", 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8669999999999998e-06 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367969+0000", 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.297e-06 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.367973+0000", 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8029999999999996e-06 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368053+0000", 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0068999999999995e-05 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.368057+0000", 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6459999999999999e-06 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729212+0000", 2024-09-13T13:52:46.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36115478099999998 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.729234+0000", 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1837000000000001e-05 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756310+0000", 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027075881 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.756373+0000", 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3782999999999996e-05 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.757252+0000", 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087888599999999995 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1277 4.4 4:22f57a3c:::smithi15938319-42:head [write 712630~452293] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.28402112200001, 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38502278299999998, 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1277 2024-09-13T13:52:46.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374323+0000", 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374326+0000", 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.21e-06 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374880+0000", 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00055436500000000002 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374892+0000", 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-09-13T13:52:46.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374900+0000", 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4800000000000004e-06 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374945+0000", 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5503000000000001e-05 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.762 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.374952+0000", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728551+0000", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35359882599999998 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728640+0000", 2024-09-13T13:52:46.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8653000000000007e-05 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740412+0000", 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011772512000000001 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740486+0000", 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3738000000000004e-05 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748146+0000", 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076603620000000004 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748254+0000", 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107427 2024-09-13T13:52:46.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.748452+0000", 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000197856 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758807+0000", 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010355493 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758889+0000", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1643000000000004e-05 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759216+0000", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032695199999999998 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759346+0000", 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013056999999999999 2024-09-13T13:52:46.767 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1278 4.4 4:22f57a3c:::smithi15938319-42:head [write 1583497~499693] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 138.27906092800001, 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38090608999999997, 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:46.768 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1278 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:46.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379284+0000", 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9500000000000001e-07 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379711+0000", 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042710099999999999 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379723+0000", 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781e-05 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379746+0000", 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2310000000000002e-05 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379801+0000", 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5569000000000003e-05 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.379807+0000", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6849999999999998e-06 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728860+0000", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34905329499999999 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.728890+0000", 2024-09-13T13:52:46.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0124000000000002e-05 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740535+0000", 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011644832000000001 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.740588+0000", 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3353000000000002e-05 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750605+0000", 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010016199999999999 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750704+0000", 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9580000000000005e-05 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.750907+0000", 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020279099999999999 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.758857+0000", 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079501529999999997 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.759943+0000", 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001085377 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760048+0000", 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105775 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:28.760190+0000", 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014136899999999999 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1558 4.4 4:21c00763:::smithi15938319-21:head [setxattr _header (56) in=7b,truncate 3848770] snapc 9=[9,8,7,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 131.33707121099999, 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.45132517, 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.778 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1558 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321273+0000", 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321274+0000", 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:46.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321281+0000", 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.401e-06 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321296+0000", 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4245e-05 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321302+0000", 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5359999999999998e-06 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322322+0000", 2024-09-13T13:52:46.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010203090000000001 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322412+0000", 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9472e-05 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5", 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322567+0000", 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155377 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.323062+0000", 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00049479000000000005 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772504+0000", 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44944161999999999 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772541+0000", 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7321999999999999e-05 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772598+0000", 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7388e-05 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.785 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1559 4.4 4:21c00763:::smithi15938319-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e59)", 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 131.33703207299999, 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.451447606, 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1559 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321312+0000", 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321313+0000", 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.68e-07 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321316+0000", 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9129999999999999e-06 2024-09-13T13:52:46.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321324+0000", 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0220000000000004e-06 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.321330+0000", 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6559999999999997e-06 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322827+0000", 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014974299999999999 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.322905+0000", 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7768999999999994e-05 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772635+0000", 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.44973000699999999 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772694+0000", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8736000000000001e-05 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:35.772760+0000", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6505000000000001e-05 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2574 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:46.791 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 104.79560207599999, 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95062650900000001, 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2574 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862742+0000", 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862744+0000", 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9750000000000001e-06 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862756+0000", 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1871999999999999e-05 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862773+0000", 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6682999999999999e-05 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862783+0000", 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7210000000000006e-06 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6966999999999997e-05 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862868+0000", 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4090000000000003e-06 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813360+0000", 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049170999999999 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813369+0000", 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1719999999999996e-06 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.797 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2575 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 104.79553912999999, 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.950597727, 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.798 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2575 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862805+0000", 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862806+0000", 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7199999999999999e-07 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862809+0000", 2024-09-13T13:52:46.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5899999999999999e-06 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862823+0000", 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3427e-05 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862829+0000", 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862896+0000", 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6252000000000002e-05 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862901+0000", 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3850000000000001e-06 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813394+0000", 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95049274699999997 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813403+0000", 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3500000000000003e-06 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:2576 4.2 4.9c041cb2 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:46.804 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 104.795505448, 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95061743300000001, 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2576 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862839+0000", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862840+0000", 2024-09-13T13:52:46.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6100000000000001e-07 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862842+0000", 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8660000000000002e-06 2024-09-13T13:52:46.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862855+0000", 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359e-05 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862860+0000", 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4439999999999999e-06 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862916+0000", 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6053e-05 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.862919+0000", 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8959999999999999e-06 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813447+0000", 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95052775899999997 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:02.813456+0000", 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5950000000000005e-06 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.810 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.811 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.811 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:46.811 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:46.811 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:46.812 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:46.813 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:46.813 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:46.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:46.815 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:46.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:46.815 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:46.811+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:46.914 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:46.912+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:46.957 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:46.957 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:46.957 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:46.957 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:46.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:47.015 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:47.013+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:47.068 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:47.068 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:47.068 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:47.068 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:47.069 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:47.077 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:47.115 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:47.113+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:201 3.3 3:f9634e5d:::smithi15938319-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.44280111699999, 2024-09-13T13:52:47.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029331221000000001, 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737116+0000", 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737117+0000", 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-09-13T13:52:47.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737134+0000", 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6511000000000002e-05 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737141+0000", 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0779999999999998e-06 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737147+0000", 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6120000000000004e-06 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737203+0000", 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5878000000000003e-05 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.737328+0000", 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124263 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746042+0000", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087146129999999995 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756535+0000", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010492573 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766333+0000", 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097985829999999996 2024-09-13T13:52:47.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766376+0000", 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2781999999999998e-05 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.766447+0000", 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0961e-05 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:202 3.3 3:e1991250:::smithi15938319-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.43756049500001, 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029401630000000002, 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.189 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 202 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742356+0000", 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742357+0000", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.92e-07 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742362+0000", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742392+0000", 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9760999999999998e-05 2024-09-13T13:52:47.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742397+0000", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742489+0000", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1577999999999996e-05 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742694+0000", 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000205546 2024-09-13T13:52:47.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.760087+0000", 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017393101000000001 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771476+0000", 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011388178000000001 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771641+0000", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016581899999999999 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771709+0000", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7237999999999996e-05 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771758+0000", 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9468000000000001e-05 2024-09-13T13:52:47.194 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:203 3.3 3:f4bd0f71:::smithi15938319-4 low_tier:head [copy-from ver 6] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.437509181, 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051397365, 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 203 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742408+0000", 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9699999999999996e-07 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742440+0000", 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.188e-05 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742448+0000", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8869999999999992e-06 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742454+0000", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5940000000000001e-06 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742924+0000", 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047049899999999998 2024-09-13T13:52:47.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743025+0000", 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101127 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.764912+0000", 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021886749 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771884+0000", 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069716999999999999 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793726+0000", 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021842572000000001 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793764+0000", 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7373999999999998e-05 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.793805+0000", 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1486000000000002e-05 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:209 3.3 3:c3255dc7:::smithi15938319-10 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.43743698399999, 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02745233, 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.201 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742480+0000", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3600000000000004e-07 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742484+0000", 2024-09-13T13:52:47.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1080000000000001e-06 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742490+0000", 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742496+0000", 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743133+0000", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063746500000000004 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743220+0000", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7083000000000006e-05 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.757147+0000", 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013926676000000001 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767827+0000", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010680263000000001 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769843+0000", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020157980000000001 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769895+0000", 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.189e-05 2024-09-13T13:52:47.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769932+0000", 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7537999999999999e-05 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:211 3.3 3:c2651e8a:::smithi15938319-12 low_tier:head [copy-from ver 3] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.437397979, 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031951795999999998, 2024-09-13T13:52:47.207 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 211 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742519+0000", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742520+0000", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1299999999999999e-07 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742525+0000", 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8289999999999998e-06 2024-09-13T13:52:47.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742532+0000", 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9789999999999996e-06 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742539+0000", 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0620000000000003e-06 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743356+0000", 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000817598 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743485+0000", 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012903599999999999 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.761648+0000", 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018162654 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771777+0000", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010129123 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774404+0000", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002627416 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774440+0000", 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5182999999999998e-05 2024-09-13T13:52:47.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774471+0000", 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1102999999999997e-05 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:215 3.3 3:f73dc719:::smithi15938319-16 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.437333567, 2024-09-13T13:52:47.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032029583, 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 215 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742583+0000", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742584+0000", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.72e-07 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742586+0000", 2024-09-13T13:52:47.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2139999999999999e-06 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742590+0000", 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9199999999999997e-06 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.742593+0000", 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7e-06 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743833+0000", 2024-09-13T13:52:47.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012407340000000001 2024-09-13T13:52:47.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743920+0000", 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6593000000000003e-05 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.763288+0000", 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019368515999999999 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.771783+0000", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084941610000000001 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774515+0000", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027323320000000001 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774570+0000", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4993e-05 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774613+0000", 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3047999999999998e-05 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:221 3.3 3:d9f1a966:::smithi15938319-22 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.41198347100001, 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044481038000000001, 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.222 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 221 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767933+0000", 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767935+0000", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3519999999999999e-06 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767941+0000", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0979999999999999e-06 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767951+0000", 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0241000000000001e-05 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767959+0000", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8280000000000003e-06 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768009+0000", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9719000000000003e-05 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768124+0000", 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115178 2024-09-13T13:52:47.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.782191+0000", 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014066944 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796459+0000", 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014268271000000001 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812349+0000", 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015890013000000001 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812384+0000", 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5200999999999997e-05 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.812415+0000", 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0193e-05 2024-09-13T13:52:47.227 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:236 3.3 3:cf957454:::smithi15938319-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.38142264699999, 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046989635000000002, 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.228 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798494+0000", 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798495+0000", 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.37e-07 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798498+0000", 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0359999999999998e-06 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798504+0000", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2489999999999998e-06 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798507+0000", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3390000000000001e-06 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798612+0000", 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010435 2024-09-13T13:52:47.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.798697+0000", 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5492000000000001e-05 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.805547+0000", 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006849412 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.810694+0000", 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051473659999999996 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845371+0000", 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034676719000000002 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845434+0000", 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3455999999999997e-05 2024-09-13T13:52:47.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845484+0000", 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9679000000000003e-05 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.4571.0:236 3.3 e41/33 ondisk, result = 0)", 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.36009605999999, 2024-09-13T13:52:47.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025668758999999999, 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819821+0000", 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819823+0000", 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0020000000000001e-06 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.819836+0000", 2024-09-13T13:52:47.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2828e-05 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845268+0000", 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.025432084000000001 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845279+0000", 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1205e-05 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845353+0000", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4336000000000001e-05 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845364+0000", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0988999999999999e-05 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.845490+0000", 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125315 2024-09-13T13:52:47.238 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:246 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.28187978299999, 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87099432899999996, 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.239 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 246 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898037+0000", 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898039+0000", 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898055+0000", 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739999999999998e-05 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898068+0000", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3139000000000001e-05 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898077+0000", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7790000000000007e-06 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898166+0000", 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8158000000000005e-05 2024-09-13T13:52:47.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.898327+0000", 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000161096 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.759670+0000", 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86134357399999995 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.764259+0000", 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045886369999999996 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768958+0000", 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046993750000000004 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.768996+0000", 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7231999999999999e-05 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.769031+0000", 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5982999999999998e-05 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1793 4.5 4:a9f43e9a:::smithi15938319-36:b [read 1988192~247139,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.937908123, 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39578151, 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.245 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1793 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242009+0000", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242010+0000", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5069999999999999e-06 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242017+0000", 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9750000000000001e-06 2024-09-13T13:52:47.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242030+0000", 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2661e-05 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242041+0000", 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242160+0000", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119382 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242169+0000", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1160000000000005e-06 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245486+0000", 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033170389999999999 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245496+0000", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5480000000000007e-06 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636773+0000", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39127700999999998 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.636971+0000", 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019797900000000001 2024-09-13T13:52:47.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637790+0000", 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00081952000000000001 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1794 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.937850071, 2024-09-13T13:52:47.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39816452000000002, 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1794 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:47.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242067+0000", 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4899999999999995e-07 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242071+0000", 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5769999999999998e-06 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242077+0000", 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3080000000000004e-06 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242083+0000", 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4920000000000001e-06 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242243+0000", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000159999 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242249+0000", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3269999999999997e-06 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245607+0000", 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033582769999999998 2024-09-13T13:52:47.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245617+0000", 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2289999999999992e-06 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.637997+0000", 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39237994599999998 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.638096+0000", 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9423e-05 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640231+0000", 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021353930000000002 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.257 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1795 4.5 4:a9f43e9a:::smithi15938319-36:b [read 0~2235331,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.937824112, 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.399016906, 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1795 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242093+0000", 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4900000000000001e-07 2024-09-13T13:52:47.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242096+0000", 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7800000000000001e-06 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242102+0000", 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6160000000000001e-06 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242107+0000", 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.242e-06 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242293+0000", 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000186343 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242299+0000", 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9460000000000003e-06 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245642+0000", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003342696 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245649+0000", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2350000000000002e-06 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640292+0000", 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39464307300000001 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.640409+0000", 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116333 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641110+0000", 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070119300000000002 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1805 4.5 4:a1652551:::smithi15938319-33:7 [read 1652225~885599,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.937709696, 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41658547699999998, 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1805 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.264 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242207+0000", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242208+0000", 2024-09-13T13:52:47.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0100000000000004e-07 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242213+0000", 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5979999999999999e-06 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242222+0000", 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.166e-06 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242228+0000", 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5509999999999996e-06 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242329+0000", 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000100593 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242335+0000", 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245672+0000", 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00333739 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245679+0000", 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9840000000000004e-06 2024-09-13T13:52:47.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.641143+0000", 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39546348399999998 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643192+0000", 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002049202 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657329+0000", 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014136783999999999 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.657408+0000", 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9456999999999995e-05 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658793+0000", 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001384534 2024-09-13T13:52:47.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1806 4.5 4:a1652551:::smithi15938319-33:7 [sparse-read 1378138~1140902] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.937678805, 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41733668000000002, 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1806 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242238+0000", 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242239+0000", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5000000000000003e-07 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242241+0000", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6769999999999999e-06 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242247+0000", 2024-09-13T13:52:47.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9220000000000002e-06 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242252+0000", 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0289999999999996e-06 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242378+0000", 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012618699999999999 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:47.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242388+0000", 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2769999999999994e-06 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245701+0000", 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033127590000000002 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245707+0000", 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3400000000000003e-06 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643340+0000", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39763295300000001 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643479+0000", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139235 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658823+0000", 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015344381000000001 2024-09-13T13:52:47.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.658898+0000", 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4103000000000005e-05 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659575+0000", 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067726699999999995 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1807 4.5 4:a1652551:::smithi15938319-33:7 [read 198123~2339701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.93764222199999, 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.41919602, 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1807 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.278 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242275+0000", 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242276+0000", 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7199999999999997e-07 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242281+0000", 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9640000000000002e-06 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242289+0000", 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4950000000000008e-06 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242297+0000", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1839999999999999e-06 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242425+0000", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127272 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.242431+0000", 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-06 2024-09-13T13:52:47.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245727+0000", 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003295891 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.245733+0000", 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1070000000000001e-06 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643501+0000", 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.39776840099999999 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.643567+0000", 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5058e-05 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659611+0000", 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016044738999999999 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.659676+0000", 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4590000000000003e-05 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.661471+0000", 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017949070000000001 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.284 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPush(4.4 64/63 [PushOp(4:22f57a3c:::smithi15938319-42:head, version: 60'55, data_included: [606208~561152,1581056~487424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:22f57a3c:::smithi15938319-42:head@60'55, size: 3512630, copy_subset: [(0, 3512630)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2068480, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.85784735199999, 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40547752799999998, 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322070+0000", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.322071+0000", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0139999999999999e-06 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.323025+0000", 2024-09-13T13:52:47.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095392000000000003 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725926+0000", 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.40290121400000001 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.725947+0000", 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0794000000000001e-05 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726058+0000", 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111654 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.726076+0000", 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7509000000000001e-05 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.727547+0000", 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014714229999999999 2024-09-13T13:52:47.288 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(4.5 64/63 [PushReplyOp(4:a9f43e9a:::smithi15938319-36:head)])", 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.483186934, 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.185179335, 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696730+0000", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696731+0000", 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-09-13T13:52:47.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.696735+0000", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2400000000000001e-06 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878713+0000", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18197780599999999 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.878721+0000", 2024-09-13T13:52:47.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5809999999999997e-06 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881204+0000", 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024827389999999999 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.881909+0000", 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00070537799999999999 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.292 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:55 3.3 3:e5701571:::smithi15938319-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.359995411, 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25641102199999999, 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:47.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819922+0000", 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2500000000000001e-07 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819923+0000", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2750000000000001e-06 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819970+0000", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6882000000000003e-05 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.819975+0000", 2024-09-13T13:52:47.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.028e-06 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821475+0000", 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001500419 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821528+0000", 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2938e-05 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8", 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.821612+0000", 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3826999999999995e-05 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.824353+0000", 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027408010000000002 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076261+0000", 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25190818399999998 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076300+0000", 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8445000000000002e-05 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076333+0000", 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2898000000000002e-05 2024-09-13T13:52:47.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(osd.8.23:55 3.3 e84/33 ondisk, result = 0)", 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.35197511299999, 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24839742400000001, 2024-09-13T13:52:47.300 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827942+0000", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3099999999999999e-07 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.827944+0000", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7379999999999999e-06 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076204+0000", 2024-09-13T13:52:47.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.24826057700000001 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076212+0000", 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6820000000000003e-06 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076249+0000", 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6754000000000001e-05 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076256+0000", 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.076339+0000", 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2937999999999998e-05 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.304 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:47.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:47.306 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:47.305+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:47.406 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:47.406+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:47.449 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:47.449 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:47.449 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-09-13T13:52:47.449 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:47.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:47.508 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:47.506+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:47.559 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:47.560 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-09-13T13:52:47.560 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-09-13T13:52:47.560 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-09-13T13:52:47.560 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:47.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:47.609 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:47.608+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4290.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 177.44406817399999, 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18246498, 2024-09-13T13:52:47.674 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4290", 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.27:0/1126701063", 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:47.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227629+0000", 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227630+0000", 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-06 2024-09-13T13:52:47.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227636+0000", 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9950000000000002e-06 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227645+0000", 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3239999999999992e-06 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227654+0000", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0049999999999993e-06 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227713+0000", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8730000000000002e-05 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.227721+0000", 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1620000000000008e-06 2024-09-13T13:52:47.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408779+0000", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18105739300000001 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.408957+0000", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017857999999999999 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409101+0000", 2024-09-13T13:52:47.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014334300000000001 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.409637+0000", 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053653100000000003 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410032+0000", 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039499100000000003 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410064+0000", 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1621000000000001e-05 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:49:50.410094+0000", 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0256e-05 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.681 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:205 3.2 3:563ca5f5:::smithi15938319-6 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.93932999200001, 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.03750042, 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 205 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732367+0000", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732369+0000", 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-09-13T13:52:47.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732375+0000", 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7669999999999998e-06 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732389+0000", 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3978999999999999e-05 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732396+0000", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732450+0000", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3637000000000002e-05 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732537+0000", 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7340000000000001e-05 2024-09-13T13:52:47.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.748432+0000", 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015894676 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.756238+0000", 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078064850000000002 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769801+0000", 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013562638 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769829+0000", 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8045999999999999e-05 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769868+0000", 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8791999999999997e-05 2024-09-13T13:52:47.687 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:206 3.2 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.93928068, 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035378785000000003, 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.688 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 206 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732417+0000", 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4499999999999998e-07 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732420+0000", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-06 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732424+0000", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8800000000000001e-06 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732427+0000", 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8569999999999999e-06 2024-09-13T13:52:47.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732669+0000", 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024136700000000001 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732773+0000", 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104141 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.740721+0000", 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079484020000000002 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.754146+0000", 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013424500000000001 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767713+0000", 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013566975 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767756+0000", 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3306999999999997e-05 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767795+0000", 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9431000000000002e-05 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:207 3.2 3:755d2207:::smithi15938319-8 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.939264976, 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035746078000000001, 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 207 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.695 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732432+0000", 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1899999999999999e-07 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732434+0000", 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8309999999999999e-06 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732437+0000", 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8729999999999999e-06 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732440+0000", 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4439999999999998e-06 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732896+0000", 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045632399999999998 2024-09-13T13:52:47.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732995+0000", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9275000000000001e-05 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.743304+0000", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010308677 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755873+0000", 2024-09-13T13:52:47.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012569195 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768090+0000", 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012217024 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768138+0000", 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7661000000000003e-05 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.700 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.768178+0000", 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0555e-05 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:210 3.2 3:552c2741:::smithi15938319-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.9392373, 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037261093000000002, 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.701 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732460+0000", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732464+0000", 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.641e-06 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732469+0000", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7829999999999998e-06 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.732473+0000", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5680000000000001e-06 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733120+0000", 2024-09-13T13:52:47.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064688600000000005 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.733233+0000", 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112621 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.746163+0000", 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012929964 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.755997+0000", 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0098341990000000001 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769668+0000", 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013670687000000001 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769697+0000", 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9680999999999999e-05 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.769721+0000", 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3711999999999999e-05 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.707 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:219 3.2 3:650cb820:::smithi15938319-20 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.904019702, 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041197026999999997, 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.708 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 219 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767677+0000", 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767678+0000", 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767683+0000", 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5750000000000002e-06 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767691+0000", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3939999999999996e-06 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767698+0000", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2729999999999999e-06 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767826+0000", 2024-09-13T13:52:47.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000128777 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.767942+0000", 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011544699999999999 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.781336+0000", 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013393950999999999 2024-09-13T13:52:47.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796232+0000", 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014895992 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808782+0000", 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012549945 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808823+0000", 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1099000000000001e-05 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.808875+0000", 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1598999999999998e-05 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:227 3.2 3:626302dd:::smithi15938319-28 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.89695072000001, 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039879611000000002, 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 227 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.715 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774746+0000", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774748+0000", 2024-09-13T13:52:47.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5409999999999999e-06 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774753+0000", 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8590000000000004e-06 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774761+0000", 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3110000000000006e-06 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.774769+0000", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2509999999999994e-06 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776177+0000", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014079780000000001 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.776283+0000", 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010561900000000001 2024-09-13T13:52:47.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.784157+0000", 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078739680000000003 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.796504+0000", 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012346733 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814471+0000", 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017967385999999998 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814571+0000", 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9741000000000006e-05 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.814626+0000", 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5223999999999998e-05 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.720 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:237 3.2 3:78d93fd4:::smithi15938319-38 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.86241225800001, 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039617250999999999, 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809285+0000", 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809286+0000", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0750000000000001e-06 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809291+0000", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9180000000000002e-06 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809302+0000", 2024-09-13T13:52:47.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0781000000000001e-05 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809309+0000", 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809347+0000", 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8232999999999998e-05 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.809460+0000", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112454 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.821497+0000", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012037946000000001 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.824925+0000", 2024-09-13T13:52:47.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034275730000000002 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848769+0000", 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023844430999999999 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848866+0000", 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6248000000000004e-05 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.848902+0000", 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6465999999999999e-05 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:243 3.2 3:6893a2ac:::smithi15938319-44 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.85478241600001, 2024-09-13T13:52:47.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035105061, 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 243 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:47.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816915+0000", 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816916+0000", 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816920+0000", 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0520000000000003e-06 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816930+0000", 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0026e-05 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816935+0000", 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0150000000000003e-06 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.816977+0000", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2497000000000003e-05 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.817064+0000", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6453999999999996e-05 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.828985+0000", 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011921851000000001 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.836885+0000", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078998760000000001 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851962+0000", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015076905 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851994+0000", 2024-09-13T13:52:47.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2140000000000001e-05 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.852020+0000", 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5466e-05 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:245 3.2 3:60c9a0cb:::smithi15938319-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.85140604399999, 2024-09-13T13:52:47.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031608922999999997, 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 245 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820291+0000", 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820293+0000", 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.376e-06 2024-09-13T13:52:47.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820303+0000", 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0332999999999999e-05 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820319+0000", 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6575999999999999e-05 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820328+0000", 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8359999999999998e-06 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820380+0000", 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1739e-05 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.820488+0000", 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108292 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.827316+0000", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006828059 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.833713+0000", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063961929999999997 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851831+0000", 2024-09-13T13:52:47.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018118759000000002 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851870+0000", 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8803000000000001e-05 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.851900+0000", 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9957000000000001e-05 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:279 4.b 4:d69faa47:::smithi15938319-30:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.748546764, 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63131724899999997, 2024-09-13T13:52:47.740 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 279 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923150+0000", 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923151+0000", 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-07 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923157+0000", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2000000000000002e-06 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923168+0000", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1201e-05 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923175+0000", 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1620000000000002e-06 2024-09-13T13:52:47.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923234+0000", 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8987000000000003e-05 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:12.923318+0000", 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3895999999999998e-05 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2", 2024-09-13T13:52:47.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.553651+0000", 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.63033278000000004 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554267+0000", 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00061648200000000001 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554394+0000", 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012689999999999999 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554427+0000", 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3417999999999998e-05 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:13.554468+0000", 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0333000000000002e-05 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:785 4.b 4:def5de2b:::smithi15938319-3:head [tier-promote] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.53752322400001, 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93650431700000003, 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 785 2024-09-13T13:52:47.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134174+0000", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599999999999998e-07 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134176+0000", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134181+0000", 2024-09-13T13:52:47.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4739999999999999e-06 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.134183+0000", 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9459999999999998e-06 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135258+0000", 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010742659999999999 2024-09-13T13:52:47.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.135319+0000", 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1200999999999999e-05 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070678+0000", 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93535935599999998 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "pg_update_log_missing_reply(4.b epoch 44/34 rep_tid 75 lcod 44'33)", 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.52600794099999, 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92499359400000003, 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145689+0000", 2024-09-13T13:52:47.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.49e-07 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:17.145691+0000", 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.623e-06 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070379+0000", 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92468808700000005 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070393+0000", 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.413e-05 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070547+0000", 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154019 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:18.070683+0000", 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013558600000000001 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1778 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 127.424699473, 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0188779100000001, 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.756 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1778 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246998+0000", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.246999+0000", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.759e-06 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247006+0000", 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7610000000000002e-06 2024-09-13T13:52:47.758 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247021+0000", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4867e-05 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247030+0000", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2320000000000007e-06 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:47.759 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1354999999999997e-05 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247110+0000", 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9629999999999998e-06 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248922+0000", 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018125449999999 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.760 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249070+0000", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014734500000000001 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265753+0000", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016683525000000001 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.265876+0000", 2024-09-13T13:52:47.761 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012255799999999999 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1779 4.2 4:4eae1ead:::smithi15938319-37:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 127.42463512499999, 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.019139456, 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.762 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1779 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247062+0000", 2024-09-13T13:52:47.763 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247063+0000", 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1699999999999998e-07 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247066+0000", 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8099999999999999e-06 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.764 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247073+0000", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8090000000000004e-06 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247079+0000", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247133+0000", 2024-09-13T13:52:47.765 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4131000000000001e-05 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247138+0000", 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2680000000000002e-06 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249341+0000", 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.002202756 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:47.766 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249389+0000", 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7752e-05 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266141+0000", 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016752540999999999 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266202+0000", 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100999999999999e-05 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.767 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1780 4.2 4:4eae1ead:::smithi15938319-37:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 127.424607063, 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0192686129999999, 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.768 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1780 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247090+0000", 2024-09-13T13:52:47.769 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247091+0000", 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.46e-07 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247093+0000", 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2510000000000001e-06 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.770 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247098+0000", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7979999999999996e-06 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247102+0000", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7350000000000004e-06 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247160+0000", 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7215e-05 2024-09-13T13:52:47.771 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:40.247165+0000", 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3079999999999998e-06 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249424+0000", 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0022588059999999 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.772 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249475+0000", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1069999999999997e-05 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266254+0000", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016779387999999999 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.266359+0000", 2024-09-13T13:52:47.773 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104596 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:1808 4.2 4:4d382039:::smithi15938319-28:head [write 652138~461367,stat] snapc c=[c,b,9,8,7] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 126.422925173, 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032534589000000003, 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.774 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 1808 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.775 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248772+0000", 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.248773+0000", 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3850000000000001e-06 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249345+0000", 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057147700000000003 2024-09-13T13:52:47.776 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249353+0000", 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6769999999999995e-06 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249363+0000", 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.777 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249559+0000", 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019625999999999999 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for missing object", 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.249695+0000", 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013579799999999999 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260436+0000", 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010741690999999999 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.778 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for degraded object", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.260501+0000", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4629999999999996e-05 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277499+0000", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016998271999999998 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277564+0000", 2024-09-13T13:52:47.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5129999999999995e-05 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.277691+0000", 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012630599999999999 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.280431+0000", 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002740532 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281192+0000", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00076073000000000004 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281235+0000", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2936999999999999e-05 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.281307+0000", 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1773999999999994e-05 2024-09-13T13:52:47.781 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPushReply(3.2 64/63 [PushReplyOp(3:60c9a0cb:::smithi15938319-46 low_tier:head)])", 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 125.983396965, 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.605271114, 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688300+0000", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688301+0000", 2024-09-13T13:52:47.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0130000000000001e-06 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:41.688308+0000", 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9299999999999997e-06 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292630+0000", 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60432188399999998 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292636+0000", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6350000000000003e-06 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.292688+0000", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2472000000000002e-05 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:50:42.293571+0000", 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088318000000000001 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.785 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(osd.8.23:51 3.2 3:626302dd:::smithi15938319-28 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e84)", 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.85476121000001, 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42905165200000001, 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.786 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816936+0000", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816937+0000", 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5799999999999998e-07 2024-09-13T13:52:47.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:00.816941+0000", 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9530000000000001e-06 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244439+0000", 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.42749784699999999 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244451+0000", 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.208e-05 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244544+0000", 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3277999999999997e-05 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244614+0000", 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0316999999999998e-05 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.244720+0000", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105431 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245515+0000", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079572300000000002 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245928+0000", 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041280800000000002 2024-09-13T13:52:47.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.791 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.791 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245959+0000", 2024-09-13T13:52:47.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1179000000000001e-05 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:01.245988+0000", 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8277999999999999e-05 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.4571.0:3286 3.2 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head [write 0~688128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:47.792 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 76.709504609999996, 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75188161099999995, 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "continuous": false, 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.4571", 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 3286 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:47.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962193+0000", 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962195+0000", 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9520000000000001e-06 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-09-13T13:52:47.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962948+0000", 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075297199999999997 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962965+0000", 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680999999999999e-05 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.962992+0000", 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6868000000000001e-05 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963061+0000", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9276999999999994e-05 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.963067+0000", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9710000000000001e-06 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967256+0000", 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041887900000000004 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967353+0000", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6626000000000006e-05 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.967450+0000", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7207999999999995e-05 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:30.968915+0000", 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465261 2024-09-13T13:52:47.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713892+0000", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74497643700000005 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.713989+0000", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7387999999999997e-05 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-09-13T13:51:31.714074+0000", 2024-09-13T13:52:47.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5179999999999994e-05 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-09-13T13:52:47.799 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-09-13T13:52:47.800 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:47.801 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:47.799+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:47.901 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:47.900+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:47.916 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:47.916 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:47.916 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:47.916 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:47.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:48.002 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:48.001+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:48.030 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.030 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:48.030 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:48.030 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:48.030 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.039 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:48.103 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:48.102+0000 7fe867556640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33587) UID: 0 2024-09-13T13:52:48.143 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.143 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:48.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:204 3.0 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.40937212200001, 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035234827000000003, 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.144 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 204 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732185+0000", 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732187+0000", 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5809999999999999e-06 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732200+0000", 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2899e-05 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732224+0000", 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4073000000000001e-05 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732237+0000", 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2221e-05 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732291+0000", 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4060000000000001e-05 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732421+0000", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00012984300000000001 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746788+0000", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014367029 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764355+0000", 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017567444000000002 2024-09-13T13:52:48.148 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767360+0000", 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003005078 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767393+0000", 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2920000000000003e-05 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767420+0000", 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6679e-05 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:208 3.0 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.40929472100001, 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.035349930000000002, 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.150 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 208 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732262+0000", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732263+0000", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7999999999999995e-07 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732268+0000", 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4410000000000001e-06 2024-09-13T13:52:48.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732274+0000", 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2290000000000002e-06 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732281+0000", 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2479999999999999e-06 2024-09-13T13:52:48.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732572+0000", 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029145700000000001 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732677+0000", 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010489600000000001 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749515+0000", 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016837652000000002 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766608+0000", 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017093240999999999 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767533+0000", 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00092455200000000003 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767582+0000", 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.922e-05 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767612+0000", 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0414000000000001e-05 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.156 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:212 3.0 3:2a10c46e:::smithi15938319-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.409142183, 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.031892380999999997, 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 212 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.157 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732415+0000", 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732416+0000", 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9800000000000003e-07 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732422+0000", 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9329999999999998e-06 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732428+0000", 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7410000000000003e-06 2024-09-13T13:52:48.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732433+0000", 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.335e-06 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732803+0000", 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00037054699999999999 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732882+0000", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8858999999999998e-05 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.741135+0000", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0082533239999999994 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745720+0000", 2024-09-13T13:52:48.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045844090000000002 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764094+0000", 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018374670999999999 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764253+0000", 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000158775 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.764307+0000", 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3989000000000001e-05 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:214 3.0 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.409036144, 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036890823000000003, 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.163 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 214 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732521+0000", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3900000000000005e-07 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732525+0000", 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4979999999999998e-06 2024-09-13T13:52:48.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732532+0000", 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.109e-06 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732543+0000", 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0859000000000001e-05 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.732983+0000", 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000440295 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.733102+0000", 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000118595 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751723+0000", 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.018621595000000001 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766801+0000", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015077703 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769328+0000", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002526721 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769375+0000", 2024-09-13T13:52:48.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7049000000000003e-05 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769412+0000", 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6860000000000003e-05 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:206 3.2 e41/33 3:7e87d171:::smithi15938319-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.40069706599999, 2024-09-13T13:52:48.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026673819000000001, 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:48.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.740860+0000", 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.85e-07 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745593+0000", 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047326809999999999 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745789+0000", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019638300000000001 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745799+0000", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4290000000000007e-06 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745859+0000", 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0575999999999997e-05 2024-09-13T13:52:48.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.745876+0000", 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6582000000000001e-05 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767501+0000", 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021625103 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.767534+0000", 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2879999999999997e-05 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:222 3.0 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.373330356, 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.048455966000000003, 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.174 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 222 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768227+0000", 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3e-07 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768234+0000", 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.764e-06 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768244+0000", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5289999999999997e-06 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768249+0000", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8940000000000001e-06 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768312+0000", 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3585999999999997e-05 2024-09-13T13:52:48.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.768406+0000", 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3947e-05 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790485+0000", 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022078411999999999 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816298+0000", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.025813451000000001 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816562+0000", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026391300000000001 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816630+0000", 2024-09-13T13:52:48.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7712000000000005e-05 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816683+0000", 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3028000000000001e-05 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:226 3.0 3:3fed19c5:::smithi15938319-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.37156375800001, 2024-09-13T13:52:48.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.046485152000000002, 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 226 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:48.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769993+0000", 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769994+0000", 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.8400000000000003e-07 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.769998+0000", 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1649999999999999e-06 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770006+0000", 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1999999999999994e-06 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770012+0000", 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7660000000000004e-06 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770050+0000", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8173999999999997e-05 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.770138+0000", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.7862999999999995e-05 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785535+0000", 2024-09-13T13:52:48.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015396732999999999 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.798360+0000", 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012824577 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816323+0000", 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017963403999999999 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816394+0000", 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1124999999999994e-05 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816478+0000", 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4361000000000001e-05 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:229 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.186 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.366684416, 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.041863704000000002, 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 229 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774873+0000", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774874+0000", 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.486e-06 2024-09-13T13:52:48.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774883+0000", 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1449999999999997e-06 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774898+0000", 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4628e-05 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774907+0000", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5880000000000004e-06 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.774972+0000", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4338999999999995e-05 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.775091+0000", 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0001193 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794425+0000", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.019333559 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816512+0000", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022087345000000001 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816691+0000", 2024-09-13T13:52:48.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017912000000000001 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816716+0000", 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5352999999999999e-05 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816736+0000", 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9840999999999999e-05 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.192 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:230 3.0 3:1d71fd43:::smithi15938319-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.35617269100001, 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036752807999999998, 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.193 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 230 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785384+0000", 2024-09-13T13:52:48.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785385+0000", 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.094e-06 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785390+0000", 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8609999999999997e-06 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785399+0000", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.0899999999999994e-06 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.785404+0000", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8199999999999996e-06 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792494+0000", 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0070895300000000001 2024-09-13T13:52:48.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.792632+0000", 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013831999999999999 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814683+0000", 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.022051325 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821560+0000", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0068770460000000004 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822041+0000", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00048074199999999999 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822096+0000", 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5316999999999999e-05 2024-09-13T13:52:48.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.822137+0000", 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0663e-05 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:238 3.0 3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.331915401, 2024-09-13T13:52:48.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039204231999999999, 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 238 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809642+0000", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809643+0000", 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3549999999999999e-06 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809649+0000", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.409e-06 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809661+0000", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1963e-05 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.809668+0000", 2024-09-13T13:52:48.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6069999999999996e-06 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.816841+0000", 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0071729020000000001 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817000+0000", 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00015946299999999999 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.827811+0000", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010810732 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848693+0000", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.020882369000000001 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848766+0000", 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2880999999999996e-05 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848809+0000", 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2738999999999999e-05 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848846+0000", 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6811999999999999e-05 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:239 3.0 3:1e720c31:::smithi15938319-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.329030356, 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.039542714999999999, 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 239 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.206 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812527+0000", 2024-09-13T13:52:48.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5899999999999995e-07 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812533+0000", 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4940000000000003e-06 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812543+0000", 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0536999999999999e-05 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.812549+0000", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1080000000000001e-06 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817149+0000", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046004110000000004 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817240+0000", 2024-09-13T13:52:48.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.1079e-05 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.839597+0000", 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.02235676 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848759+0000", 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0091626720000000002 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852017+0000", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0032574460000000002 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852044+0000", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6665999999999998e-05 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.852069+0000", 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5783e-05 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.211 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:240 3.0 3:3d1c96ff:::smithi15938319-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.32682633100001, 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.036747751000000002, 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.212 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 240 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814731+0000", 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7499999999999999e-07 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814739+0000", 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7710000000000006e-06 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814748+0000", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.9530000000000005e-06 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.814753+0000", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9060000000000001e-06 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817335+0000", 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0025817729999999999 2024-09-13T13:52:48.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.817514+0000", 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000179486 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.834410+0000", 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016895666 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848735+0000", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014325384 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851390+0000", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.002654338 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851439+0000", 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9449e-05 2024-09-13T13:52:48.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.851478+0000", 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9249999999999999e-05 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:237 3.2 e41/33 3:78d93fd4:::smithi15938319-38 low_tier:head v 41'8, mlcod=41'6)", 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.31983568699999, 2024-09-13T13:52:48.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.026651953999999999, 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821721+0000", 2024-09-13T13:52:48.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.821725+0000", 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9580000000000001e-06 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826449+0000", 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047236190000000001 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826484+0000", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5379000000000001e-05 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826494+0000", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0049e-05 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826564+0000", 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9424999999999993e-05 2024-09-13T13:52:48.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.826587+0000", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3569000000000001e-05 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848306+0000", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.021718597999999999 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.848373+0000", 2024-09-13T13:52:48.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7356999999999999e-05 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(osd.14.23:17 3.0 3:2ac2f6e0:::smithi15938319-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.21484407299999, 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62663421799999997, 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926713+0000", 2024-09-13T13:52:48.224 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2099999999999997e-07 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926716+0000", 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7609999999999999e-06 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551772+0000", 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.62505579499999997 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.225 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551798+0000", 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6174e-05 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.551993+0000", 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000194446 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552129+0000", 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013620900000000001 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.226 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552309+0000", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017982999999999999 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.552896+0000", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058690700000000001 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553287+0000", 2024-09-13T13:52:48.227 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00039131300000000001 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553320+0000", 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2787000000000003e-05 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:13.553347+0000", 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7475e-05 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.228 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2126 4.8 4:1b24d731:::smithi15938319-13:head [setxattr _header (57) in=7b,truncate 2806769] snapc 11=[11,10,f,e,b,8] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.39125421200001, 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74321575100000004, 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2126 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750303+0000", 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750304+0000", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.0999999999999998e-07 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750307+0000", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.6339999999999999e-06 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750313+0000", 2024-09-13T13:52:48.231 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0510000000000002e-06 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750319+0000", 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.9279999999999998e-06 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751219+0000", 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089940699999999996 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.232 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751295+0000", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6916999999999996e-05 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 11", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751415+0000", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119459 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.752578+0000", 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0011627880000000001 2024-09-13T13:52:48.233 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493430+0000", 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74085221099999998 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493471+0000", 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1155000000000002e-05 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.234 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493519+0000", 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7491000000000003e-05 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2127 4.8 4:1b24d731:::smithi15938319-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.391221422, 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74335525000000002, 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.235 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2127 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.236 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750336+0000", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0299999999999999e-07 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750339+0000", 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5409999999999999e-06 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.237 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750345+0000", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3829999999999997e-06 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750351+0000", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4570000000000002e-06 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751635+0000", 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012834260000000001 2024-09-13T13:52:48.238 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for rw locks", 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751699+0000", 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4485000000000003e-05 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493554+0000", 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74185420899999999 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.239 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493619+0000", 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5158000000000002e-05 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493691+0000", 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2088000000000005e-05 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPush(3.2 74/73 [PushOp(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head, version: 54'64, data_included: [1048576~266240], data_size: 266240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:5174431b:::a76e70f184b8ac83c1dfc0e48062a2f4a489768ac15349a7749de78ba60fcccc:head@54'64, size: 1314816, copy_subset: [(0, 1314816)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1314816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T13:52:48.240 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 117.27546191499999, 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27247468000000002, 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.241 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866095+0000", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866096+0000", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4099999999999999e-07 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.866411+0000", 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00031513500000000002 2024-09-13T13:52:48.242 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137854+0000", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.27144290500000001 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137867+0000", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3642000000000001e-05 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137966+0000", 2024-09-13T13:52:48.243 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.8889999999999997e-05 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.137980+0000", 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4351e-05 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.138570+0000", 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00058931599999999999 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.244 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:2323 3.0 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head [write 0~1985536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 112.350859226, 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.34081940500000002, 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.245 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 2323 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.248 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790698+0000", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.790699+0000", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6479999999999999e-06 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791549+0000", 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084982500000000002 2024-09-13T13:52:48.249 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791567+0000", 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7819e-05 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791577+0000", 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5550000000000008e-06 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.250 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791661+0000", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4153999999999994e-05 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.791671+0000", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0314e-05 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796204+0000", 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045329960000000001 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.251 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796286+0000", 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1941999999999996e-05 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 7", 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796369+0000", 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3423000000000004e-05 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.252 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.801027+0000", 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046575000000000002 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131403+0000", 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33037628899999999 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131464+0000", 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1179000000000005e-05 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.253 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131517+0000", 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2760999999999998e-05 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop_reply(osd.3.0:192 4.8 e106/34 ondisk, result = 0)", 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 84.400649478999995, 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080426668000000007, 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.254 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.255 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740908+0000", 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0699999999999997e-07 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.740912+0000", 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2560000000000004e-06 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.256 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821087+0000", 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.080174492999999999 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821097+0000", 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.747e-06 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.257 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821174+0000", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6921000000000004e-05 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821187+0000", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3176999999999999e-05 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:23.821334+0000", 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000147467 2024-09-13T13:52:48.258 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3286 3.2 e114/73 3:55b1b62d:::ecb0b63f2e3bf7d71978af1a8a4d63bcfe8f567c7aee774df1ec0aff6e9eaf86:head v 114'133, mlcod=105'131)", 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 77.173680187000002, 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74551292800000002, 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.259 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.967877+0000", 2024-09-13T13:52:48.260 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.73e-07 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:30.968632+0000", 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00075455699999999999 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711912+0000", 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74328065200000004 2024-09-13T13:52:48.261 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.711933+0000", 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1279000000000001e-05 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712080+0000", 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014671199999999999 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.262 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.712101+0000", 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0339999999999998e-05 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713357+0000", 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012564550000000001 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:31.713390+0000", 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2759999999999998e-05 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.264 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.264 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.264 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.264 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T13:52:48.266 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:48.267 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:48.263+0000 7f003d1e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33594) UID: 0 2024-09-13T13:52:48.366 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:48.365+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:48.410 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.410 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:48.410 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:48.410 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.419 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:48.467 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:48.466+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:48.522 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:48.522 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:48.522 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.530 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:48.568 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:48.567+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:204 3.0 e41/33 3:2bb96f16:::smithi15938319-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.88628508599999, 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013589403, 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.636 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746896+0000", 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.637 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.746898+0000", 2024-09-13T13:52:48.638 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.762e-06 2024-09-13T13:52:48.638 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.638 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.638 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.638 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748268+0000", 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001370118 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748325+0000", 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6827000000000003e-05 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748339+0000", 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.415e-05 2024-09-13T13:52:48.639 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748417+0000", 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7886999999999995e-05 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.748443+0000", 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6661000000000001e-05 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.640 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760422+0000", 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011978579 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.760485+0000", 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3418999999999997e-05 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:208 3.0 e41/33 3:302c63e5:::smithi15938319-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'3, mlcod=0'0)", 2024-09-13T13:52:48.641 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.88330847500001, 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016587873999999999, 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.642 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749872+0000", 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.749874+0000", 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7940000000000001e-06 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.643 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.644 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750826+0000", 2024-09-13T13:52:48.644 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000951715 2024-09-13T13:52:48.645 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.645 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.645 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.645 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750888+0000", 2024-09-13T13:52:48.645 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1783000000000001e-05 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750900+0000", 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2539e-05 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.646 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.750977+0000", 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.6250999999999999e-05 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751009+0000", 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2143e-05 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766414+0000", 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015405138000000001 2024-09-13T13:52:48.647 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766460+0000", 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.6511000000000002e-05 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:214 3.0 e41/33 3:34d010b6:::smithi15938319-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'4, mlcod=0'0)", 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.88133430400001, 2024-09-13T13:52:48.648 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.014768102, 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.649 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751847+0000", 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.751848+0000", 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1349999999999999e-06 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.650 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753051+0000", 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0012030789999999999 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753091+0000", 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9956999999999997e-05 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753103+0000", 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1956e-05 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.651 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753217+0000", 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011472900000000001 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.652 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.753243+0000", 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5901000000000002e-05 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766576+0000", 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.01333281 2024-09-13T13:52:48.653 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.766615+0000", 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8535000000000003e-05 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:222 3.0 e41/33 3:0c30b127:::smithi15938319-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'7, mlcod=41'4)", 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.84260378100001, 2024-09-13T13:52:48.654 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.017343109999999998, 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790577+0000", 2024-09-13T13:52:48.655 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.790578+0000", 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0499999999999999e-06 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791493+0000", 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00091511499999999998 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.656 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791546+0000", 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2549000000000001e-05 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791558+0000", 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1993e-05 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.657 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791631+0000", 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.2914e-05 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.658 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.791680+0000", 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9530999999999997e-05 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.659 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807865+0000", 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016185148999999999 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.807920+0000", 2024-09-13T13:52:48.660 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4809000000000003e-05 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:229 3.0 e41/33 3:2ac2f6e0:::smithi15938319-30 low_tier:head v 41'8, mlcod=41'4)", 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.83870249, 2024-09-13T13:52:48.661 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.016564876999999999, 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794478+0000", 2024-09-13T13:52:48.662 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.794479+0000", 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.3900000000000004e-07 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795529+0000", 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0010500349999999999 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.663 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795584+0000", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4211e-05 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795595+0000", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1268e-05 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795668+0000", 2024-09-13T13:52:48.664 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3625000000000001e-05 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.795689+0000", 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0727999999999999e-05 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811007+0000", 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015318263 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.665 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.811043+0000", 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5908000000000002e-05 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:269 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.717537502, 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.015781442, 2024-09-13T13:52:48.666 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 269 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.667 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915643+0000", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915645+0000", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.663e-06 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915652+0000", 2024-09-13T13:52:48.668 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6420000000000001e-06 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915665+0000", 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3337e-05 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915672+0000", 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7320000000000001e-06 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.669 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915729+0000", 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7574999999999997e-05 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.915805+0000", 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5216999999999995e-05 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.670 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.925804+0000", 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0099990659999999992 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.926798+0000", 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0009941590000000001 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931356+0000", 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584639999999999 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.671 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931396+0000", 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.9793999999999999e-05 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.931425+0000", 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8793e-05 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.672 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:275 4.6 4:6668b762:::smithi15938319-26:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 155.71273660700001, 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013706987, 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 275 2024-09-13T13:52:48.673 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920444+0000", 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.674 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920446+0000", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5829999999999999e-06 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920452+0000", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.4860000000000002e-06 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920469+0000", 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6201999999999998e-05 2024-09-13T13:52:48.675 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920477+0000", 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4910000000000005e-06 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920543+0000", 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6128000000000001e-05 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.676 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.920602+0000", 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8465999999999999e-05 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930474+0000", 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0098719340000000006 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.930906+0000", 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.677 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934096+0000", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031901249999999998 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934125+0000", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9572000000000001e-05 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.934151+0000", 2024-09-13T13:52:48.678 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5848000000000001e-05 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:369 4.6 4:6ce02a65:::smithi15938319-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 154.55809776999999, 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0094626669999999993, 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.679 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 369 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.680 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075083+0000", 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075085+0000", 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075093+0000", 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4489999999999999e-06 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.681 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075116+0000", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2337e-05 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075127+0000", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1161e-05 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075201+0000", 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4476999999999999e-05 2024-09-13T13:52:48.682 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.075283+0000", 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1200999999999997e-05 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079004+0000", 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00372099 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.683 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.079886+0000", 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088274299999999996 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084441+0000", 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045551020000000001 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084494+0000", 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.24e-05 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.684 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:14.084546+0000", 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.2027000000000002e-05 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:1252 4.6 4:6668b762:::smithi15938319-26:head [write 679016~701795] snapc 6=[6,5] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:48.685 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 140.378236284, 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013941904, 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 1252 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.686 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254945+0000", 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.254946+0000", 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7409999999999999e-06 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.687 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255092+0000", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00014609399999999999 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255109+0000", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7028999999999999e-05 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255117+0000", 2024-09-13T13:52:48.688 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4909999999999999e-06 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255177+0000", 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0331999999999998e-05 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255246+0000", 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8444999999999999e-05 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.689 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.255377+0000", 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013124799999999999 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.257696+0000", 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0023192410000000001 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268802+0000", 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.011105296000000001 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.690 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268836+0000", 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4306e-05 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:28.268887+0000", 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0680999999999998e-05 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.691 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "MOSDPGPushReply(3.0 64/63 [PushReplyOp(3:274b569f:::smithi15938319-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head)])", 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 127.345144928, 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45541287800000002, 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "reached pg", 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.692 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.693 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288036+0000", 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0699999999999997e-07 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.288040+0000", 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7019999999999999e-06 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742184+0000", 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.45414381300000001 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.694 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742210+0000", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6037000000000002e-05 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.742294+0000", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.4011000000000006e-05 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:41.743449+0000", 2024-09-13T13:52:48.695 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001154808 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2323 3.0 e79/73 3:16989df5:::ff22f344165e9200e301af25a6de1db64ed97a5f3a41e60e8f870bf132832aa0:head v 79'93, mlcod=75'91)", 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 112.83650467299999, 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.33454157899999998, 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.696 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.697 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.796676+0000", 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4700000000000001e-07 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:55.797508+0000", 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00083145400000000003 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126437+0000", 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.32892945499999998 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.698 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126447+0000", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2699999999999993e-06 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126498+0000", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1079999999999999e-05 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.126514+0000", 2024-09-13T13:52:48.699 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6152000000000001e-05 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131183+0000", 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046693129999999996 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:56.131218+0000", 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4708e-05 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.700 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3121 4.6 4:6668b762:::smithi15938319-26:head [write 736206~453891] snapc 1f=[1e,1c,1b,1a,17,16,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 86.722477413000007, 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010295673999999999, 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.701 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3121 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910703+0000", 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.702 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910705+0000", 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0210000000000001e-06 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910925+0000", 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00022029800000000001 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.703 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910938+0000", 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2737e-05 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.910944+0000", 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6320000000000002e-06 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911005+0000", 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0547999999999997e-05 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.704 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.911013+0000", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9200000000000004e-06 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915868+0000", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048554749999999997 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.915942+0000", 2024-09-13T13:52:48.705 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3616999999999997e-05 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.916074+0000", 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013179000000000001 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920046+0000", 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0039720129999999999 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.706 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920930+0000", 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00088461699999999998 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920965+0000", 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.5309999999999999e-05 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:21.920999+0000", 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3695999999999997e-05 2024-09-13T13:52:48.707 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:3177 4.6 4:6ce02a65:::smithi15938319-20:head [append 0~0] snapc 1f=[1e,1c,1b,1a,17,14,13,12] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 85.724875053999995, 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010598771999999999, 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.708 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 3177 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.709 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908306+0000", 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908309+0000", 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4789999999999999e-06 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.710 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908315+0000", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999998e-06 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908321+0000", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.485e-06 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908420+0000", 2024-09-13T13:52:48.711 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.9528e-05 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.908426+0000", 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7910000000000001e-06 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913787+0000", 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0053607780000000001 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.712 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.913948+0000", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00016093300000000001 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914121+0000", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00017322299999999999 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.914964+0000", 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00084322900000000003 2024-09-13T13:52:48.713 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918790+0000", 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0038258319999999999 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918846+0000", 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6159000000000002e-05 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.714 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:22.918905+0000", 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8187999999999999e-05 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4028 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.562333268000003, 2024-09-13T13:52:48.715 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010533003000000001, 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4028 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:48.716 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070848+0000", 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070849+0000", 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6199999999999999e-06 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.717 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070861+0000", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1973999999999999e-05 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070881+0000", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0095999999999999e-05 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070894+0000", 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2204999999999999e-05 2024-09-13T13:52:48.718 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070965+0000", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1613999999999996e-05 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070971+0000", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.395e-06 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077011+0000", 2024-09-13T13:52:48.719 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006040035 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.077114+0000", 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103727 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081381+0000", 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042663370000000003 2024-09-13T13:52:48.720 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4029 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.562249223999999, 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012297313000000001, 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.721 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4029 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.722 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070932+0000", 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7400000000000003e-07 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070935+0000", 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1089999999999998e-06 2024-09-13T13:52:48.723 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070941+0000", 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7799999999999997e-06 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070945+0000", 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2980000000000002e-06 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.724 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071034+0000", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8661999999999999e-05 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071041+0000", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5189999999999998e-06 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081442+0000", 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.010401216 2024-09-13T13:52:48.725 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.081561+0000", 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000119207 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083229+0000", 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.001667948 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.726 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4030 4.6 4:6ce02a65:::smithi15938319-20:head [sparse-read 0~1592192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139)", 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 47.562228109000003, 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722845, 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.727 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4030 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.728 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070953+0000", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2800000000000003e-07 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070955+0000", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.5060000000000001e-06 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070958+0000", 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3160000000000001e-06 2024-09-13T13:52:48.729 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.070963+0000", 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8500000000000002e-06 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071060+0000", 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.7565999999999994e-05 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.730 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.071063+0000", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0479999999999999e-06 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083261+0000", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012197846 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083377+0000", 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011555100000000001 2024-09-13T13:52:48.731 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:01.083676+0000", 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00029883399999999999 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4357 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:48.732 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.880361460000003, 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012624672, 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4357 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.733 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752819+0000", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752820+0000", 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1399999999999997e-07 2024-09-13T13:52:48.734 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752827+0000", 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.5010000000000001e-06 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752834+0000", 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7079999999999997e-06 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.735 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752838+0000", 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1110000000000001e-06 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752943+0000", 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00010495199999999999 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752947+0000", 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8789999999999996e-06 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.736 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762157+0000", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0092093769999999995 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.762260+0000", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103145 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765444+0000", 2024-09-13T13:52:48.737 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0031843850000000001 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4358 4.6 4:6ce02a65:::smithi15938319-20:head [read 0~2554697,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 35.880317814999998, 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.013684981000000001, 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.738 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4358 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.739 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752863+0000", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752864+0000", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4700000000000001e-07 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752867+0000", 2024-09-13T13:52:48.740 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4350000000000001e-06 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752874+0000", 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3869999999999997e-06 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752880+0000", 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6200000000000004e-06 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.741 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752994+0000", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011415 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.752999+0000", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.0270000000000003e-06 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765566+0000", 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012566623000000001 2024-09-13T13:52:48.742 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.765652+0000", 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.6439000000000004e-05 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.766548+0000", 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000895753 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.743 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4422 4.6 4:6ce02a65:::smithi15938319-20:head [tier-promote] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 31.712944937, 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012722221000000001, 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "started", 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.744 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4422 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920236+0000", 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.745 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920238+0000", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.618e-06 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920244+0000", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3929999999999996e-06 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920260+0000", 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6308e-05 2024-09-13T13:52:48.746 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920270+0000", 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.6490000000000003e-06 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920344+0000", 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3979000000000004e-05 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op must wait for map", 2024-09-13T13:52:48.747 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920351+0000", 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9369999999999998e-06 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924882+0000", 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045310389999999997 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.924943+0000", 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0618000000000001e-05 2024-09-13T13:52:48.748 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.932958+0000", 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0080156800000000007 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_op(client.4571.0:4423 4.6 4:6668b762:::smithi15938319-26:head [stat,delete] snapc 35=[35,34,33,32,31,2d,24,23,22,14] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 31.712882261000001, 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.012827221, 2024-09-13T13:52:48.749 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client_info": { 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client": "client.4571", 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "client_addr": "172.21.15.159:0/4173298062", 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "tid": 4423 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.750 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920299+0000", 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9999999999999998e-07 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:48.751 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920302+0000", 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6089999999999999e-06 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920309+0000", 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.844e-06 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920314+0000", 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.259e-06 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.752 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920407+0000", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.3058000000000002e-05 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting_for_map not empty", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.920414+0000", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4159999999999998e-06 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925431+0000", 2024-09-13T13:52:48.753 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005016553 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925484+0000", 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3507000000000001e-05 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "waiting for subops from 13", 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.925679+0000", 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00019484 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "op_commit", 2024-09-13T13:52:48.754 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.928735+0000", 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0030555719999999999 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "sub_op_commit_rec", 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933007+0000", 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042717700000000003 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933071+0000", 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3935000000000004e-05 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:48.755 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.933126+0000", 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5358000000000001e-05 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:48.756 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.757 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:48.756+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:48.758 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:48.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T13:52:48.862 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:48.861+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:48.862 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:48.862 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:48.862 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.871 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:48.949 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0: {} 2024-09-13T13:52:48.950 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T13:52:48.959 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:48.957+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:48.971 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1: {} 2024-09-13T13:52:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T13:52:48.975 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:48.975 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:48.975 INFO:teuthology.orchestra.run.smithi159.stdout: "complaint_time": 30, 2024-09-13T13:52:48.975 INFO:teuthology.orchestra.run.smithi159.stdout: "num_blocked_ops": 0 2024-09-13T13:52:48.976 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:48.984 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:48.994 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2: {} 2024-09-13T13:52:48.994 INFO:teuthology.orchestra.run.smithi027.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T13:52:49.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3: {} 2024-09-13T13:52:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T13:52:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-13T13:52:49.037 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5: {} 2024-09-13T13:52:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T13:52:49.059 INFO:teuthology.orchestra.run.smithi027.stdout:osd.6: {} 2024-09-13T13:52:49.059 INFO:teuthology.orchestra.run.smithi027.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T13:52:49.060 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:49.058+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:49.080 INFO:teuthology.orchestra.run.smithi027.stdout:osd.7: {} 2024-09-13T13:52:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T13:52:49.100 INFO:teuthology.orchestra.run.smithi027.stdout:osd.8: {} 2024-09-13T13:52:49.101 INFO:teuthology.orchestra.run.smithi027.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T13:52:49.122 INFO:teuthology.orchestra.run.smithi027.stdout:osd.9: {} 2024-09-13T13:52:49.122 INFO:teuthology.orchestra.run.smithi027.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T13:52:49.127 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:49.127 INFO:teuthology.orchestra.run.smithi159.stdout: "size": 20, 2024-09-13T13:52:49.127 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 600, 2024-09-13T13:52:49.127 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [ 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:49 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'1, mlcod=0'0)", 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 156.864310004, 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0049162030000000001, 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.128 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261735+0000", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.261736+0000", 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.22e-07 2024-09-13T13:52:49.129 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262154+0000", 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000418074 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262182+0000", 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.8657999999999999e-05 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.130 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262192+0000", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4250000000000004e-06 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262243+0000", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1622999999999997e-05 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262257+0000", 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3769e-05 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.131 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266593+0000", 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004335518 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266652+0000", 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.8814000000000003e-05 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.132 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:50 4.8 e41/34 4:1b24d731:::smithi15938319-13:head v 41'2, mlcod=0'0)", 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 156.86335650500001, 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041645270000000003, 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.133 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262689+0000", 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6499999999999999e-07 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.134 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262948+0000", 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00025821999999999998 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262973+0000", 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5389e-05 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.262978+0000", 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1490000000000002e-06 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.135 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263024+0000", 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5736999999999999e-05 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.263043+0000", 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.929e-05 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.136 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266808+0000", 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.003764472 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:12.266853+0000", 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5704999999999997e-05 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.137 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:1025 4.8 e48/34 4:165e03eb:::smithi15938319-32:head v 48'14, mlcod=42'11)", 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 145.35252024499999, 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0046637809999999997, 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.138 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773525+0000", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.6899999999999999e-07 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.773975+0000", 2024-09-13T13:52:49.139 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00044964400000000002 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774013+0000", 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.8510000000000002e-05 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774024+0000", 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0149000000000001e-05 2024-09-13T13:52:49.140 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774085+0000", 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1125999999999997e-05 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.774109+0000", 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4059000000000001e-05 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.141 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778140+0000", 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040313060000000001 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:23.778189+0000", 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8817999999999999e-05 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.142 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2050 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'55, mlcod=66'48)", 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 121.799609672, 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063994509999999996, 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326436+0000", 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326437+0000", 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1399999999999996e-07 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.144 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326702+0000", 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00026541500000000002 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326734+0000", 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.2212000000000003e-05 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.145 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326742+0000", 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.7360000000000001e-06 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326789+0000", 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7092999999999998e-05 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.326802+0000", 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2544000000000001e-05 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.146 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332791+0000", 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0059898850000000003 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332835+0000", 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3751999999999997e-05 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.147 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2051 4.8 e70/34 4:165e03eb:::smithi15938319-32:head v 70'56, mlcod=66'48)", 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 121.79886078299999, 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056809010000000004, 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:49.148 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327185+0000", 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1900000000000002e-07 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.149 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327193+0000", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4329999999999998e-06 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327223+0000", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.01e-05 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327230+0000", 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.486e-06 2024-09-13T13:52:49.150 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327576+0000", 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00034597399999999998 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.327588+0000", 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2160999999999999e-05 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.151 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332840+0000", 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0052517939999999997 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:47.332866+0000", 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5333999999999998e-05 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2124 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'61, mlcod=73'58)", 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:49.152 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.380526601, 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0097188889999999997, 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.153 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745519+0000", 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745521+0000", 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7799999999999999e-06 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.745657+0000", 2024-09-13T13:52:49.154 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00013646700000000001 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751490+0000", 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058329280000000002 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751505+0000", 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4674999999999999e-05 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.155 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.3619000000000001e-05 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751606+0000", 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.7311e-05 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.156 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755196+0000", 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0035906739999999999 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.755238+0000", 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1434999999999999e-05 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.157 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2125 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'62, mlcod=73'58)", 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.37539824300001, 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0073962259999999997, 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:49.158 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750647+0000", 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.750649+0000", 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.55e-06 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.159 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751461+0000", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00081191300000000004 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751578+0000", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00011694899999999999 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751583+0000", 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6099999999999997e-06 2024-09-13T13:52:49.160 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754481+0000", 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0028974270000000002 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.754504+0000", 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3343999999999999e-05 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.161 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.757996+0000", 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0034919560000000001 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.758043+0000", 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7477e-05 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.162 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2126 4.8 e74/34 4:1b24d731:::smithi15938319-13:head v 74'63, mlcod=73'58)", 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 118.374486687, 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74173815200000004, 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.163 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751559+0000", 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751560+0000", 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.0559999999999999e-06 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.164 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:50.751565+0000", 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3820000000000003e-06 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492733+0000", 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.74116788 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.165 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492741+0000", 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5619999999999998e-06 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492810+0000", 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8978000000000003e-05 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.492827+0000", 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7778000000000002e-05 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.166 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493259+0000", 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00043215199999999999 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:50:51.493297+0000", 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7363999999999997e-05 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2560 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'83, mlcod=84'79)", 2024-09-13T13:52:49.167 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 107.277126255, 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048035650000000001, 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.168 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848919+0000", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.848920+0000", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8899999999999998e-07 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849015+0000", 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.4949000000000002e-05 2024-09-13T13:52:49.169 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849049+0000", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4199000000000002e-05 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849056+0000", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8719999999999996e-06 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849109+0000", 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3368999999999997e-05 2024-09-13T13:52:49.170 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849123+0000", 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3382e-05 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853681+0000", 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045584370000000003 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.171 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853723+0000", 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.1467999999999997e-05 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2561 4.8 e85/34 4:1b24d731:::smithi15938319-13:head v 85'84, mlcod=84'79)", 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 107.276358207, 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042765060000000002, 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.172 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849687+0000", 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.173 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849688+0000", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8699999999999995e-07 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849692+0000", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2729999999999996e-06 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849732+0000", 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0373000000000001e-05 2024-09-13T13:52:49.174 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849739+0000", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.9959999999999996e-06 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849811+0000", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1392999999999999e-05 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.849834+0000", 2024-09-13T13:52:49.175 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3332999999999998e-05 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853929+0000", 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0040950179999999997 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:01.853964+0000", 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.4632999999999999e-05 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.176 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:2684 4.8 e90/34 4:165e03eb:::smithi15938319-32:head v 90'93, mlcod=90'90)", 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 101.92004492700001, 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056341009999999999, 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.177 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206000+0000", 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206001+0000", 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9599999999999999e-07 2024-09-13T13:52:49.178 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206006+0000", 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4229999999999996e-06 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206033+0000", 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6730999999999999e-05 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.179 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206038+0000", 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.5950000000000001e-06 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206085+0000", 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7756000000000002e-05 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.206099+0000", 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3427999999999999e-05 2024-09-13T13:52:49.180 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211583+0000", 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054838839999999996 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:07.211635+0000", 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1788e-05 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.181 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3456 4.3 e118/34 4:c1ae59bf:::smithi15938319-45:head v 118'27, mlcod=63'23)", 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 71.712637662000006, 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063236159999999998, 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:49.182 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413408+0000", 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413409+0000", 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.2900000000000002e-07 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.183 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413414+0000", 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4210000000000003e-06 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413439+0000", 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.5316e-05 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.413445+0000", 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.7729999999999996e-06 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.184 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414189+0000", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00074398200000000002 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.414204+0000", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.4645e-05 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419678+0000", 2024-09-13T13:52:49.185 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0054737930000000002 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:37.419731+0000", 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3757e-05 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3517 4.8 e120/34 4:165e03eb:::smithi15938319-32:head v 120'129, mlcod=118'125)", 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:49.186 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 68.819952682999997, 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0042950089999999998, 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:49.187 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306093+0000", 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.6599999999999996e-07 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306099+0000", 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1090000000000003e-06 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.188 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306129+0000", 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9955999999999999e-05 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306138+0000", 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.259e-06 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.189 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306181+0000", 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.3059999999999998e-05 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.306200+0000", 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9383999999999999e-05 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310350+0000", 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0041501899999999998 2024-09-13T13:52:49.190 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:40.310388+0000", 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.7385000000000002e-05 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:3654 4.8 e127/34 4:1b24d731:::smithi15938319-13:head v 127'136, mlcod=127'132)", 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 62.168361032, 2024-09-13T13:52:49.191 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047562300000000002, 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957684+0000", 2024-09-13T13:52:49.192 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957685+0000", 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.4000000000000001e-07 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957693+0000", 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.943e-06 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.193 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957723+0000", 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.9716000000000001e-05 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957730+0000", 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.7560000000000002e-06 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957781+0000", 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.1143000000000001e-05 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.194 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.957805+0000", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.3884999999999999e-05 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962396+0000", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004591866 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:51:46.962441+0000", 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.4181000000000003e-05 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.195 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:256 4.8 e150/34 4:165e03eb:::smithi15938319-32:2c v 150'186, mlcod=148'183)", 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.369049994999997, 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045821789999999996, 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:49.196 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756995+0000", 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.756996+0000", 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.98e-07 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.197 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757009+0000", 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.2948e-05 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757049+0000", 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0793000000000001e-05 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757058+0000", 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.8380000000000008e-06 2024-09-13T13:52:49.198 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757115+0000", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.6483000000000001e-05 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.757139+0000", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4167000000000001e-05 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761524+0000", 2024-09-13T13:52:49.199 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004384888 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.761578+0000", 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.3863999999999998e-05 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:257 4.8 e150/34 4:1b24d731:::smithi15938319-13:2c v 150'188, mlcod=148'183)", 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 36.368009307999998, 2024-09-13T13:52:49.200 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0063443070000000004, 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758036+0000", 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.201 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758037+0000", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.9999999999999996e-07 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758044+0000", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.759e-06 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758116+0000", 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.1913000000000001e-05 2024-09-13T13:52:49.202 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758123+0000", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.763e-06 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758187+0000", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.3283999999999999e-05 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.758211+0000", 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.4728000000000001e-05 2024-09-13T13:52:49.203 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764325+0000", 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.006114113 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:12.764380+0000", 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.4947000000000001e-05 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.204 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:262 4.8 e154/34 4:1b24d731:::smithi15938319-13:26 v 154'194, mlcod=150'186)", 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 32.360067272000002, 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0058817069999999999, 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:49.205 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765978+0000", 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765979+0000", 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.1999999999999999e-07 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.206 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.765984+0000", 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.9409999999999998e-06 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766015+0000", 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.1711000000000002e-05 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.207 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766021+0000", 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.0009999999999998e-06 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766125+0000", 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.000103187 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.766138+0000", 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.3067999999999999e-05 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.208 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771829+0000", 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0056913909999999996 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:16.771860+0000", 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.0787999999999997e-05 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.209 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(osd.3.0:277 4.8 e163/34 4:1b24d731:::smithi15938319-13:26 v 163'204, mlcod=157'198)", 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.944853856000002, 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0062937430000000001, 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.210 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181192+0000", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 8.1500000000000003e-07 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181200+0000", 2024-09-13T13:52:49.211 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 7.5780000000000001e-06 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181242+0000", 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.2565999999999999e-05 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.181254+0000", 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.1151999999999999e-05 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.212 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182146+0000", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.00089234399999999999 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182165+0000", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.9293e-05 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187445+0000", 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.005279265 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.213 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187485+0000", 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.0729999999999998e-05 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4604 4.8 e163/34 4:165e03eb:::smithi15938319-32:head v 163'205, mlcod=157'198)", 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 21.943512281, 2024-09-13T13:52:49.214 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.004977637, 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182533+0000", 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.215 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182534+0000", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 6.8500000000000001e-07 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182540+0000", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.959e-06 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182574+0000", 2024-09-13T13:52:49.216 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.3806999999999997e-05 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182579+0000", 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.835e-06 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182627+0000", 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.7926000000000002e-05 2024-09-13T13:52:49.217 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.182645+0000", 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 1.7419e-05 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187488+0000", 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0048430740000000002 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.218 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:27.187511+0000", 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.2932e-05 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "description": "osd_repop(client.4571.0:4654 4.3 e168/34 4:c1ae59bf:::smithi15938319-45:head v 168'66, mlcod=163'62)", 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "initiated_at": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "age": 16.924540117999999, 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0047873990000000003, 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "continuous": false, 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "type_data": { 2024-09-13T13:52:49.219 INFO:teuthology.orchestra.run.smithi159.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "events": [ 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "initiated", 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "header_read", 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.220 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "throttled", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201505+0000", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.1e-07 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "all_read", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201511+0000", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 5.5430000000000001e-06 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "dispatched", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201546+0000", 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 3.447e-05 2024-09-13T13:52:49.221 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "queued_for_pg", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201572+0000", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.6237e-05 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "reached_pg", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201667+0000", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 9.5329999999999997e-05 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "started", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.201688+0000", 2024-09-13T13:52:49.222 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 2.0568999999999999e-05 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "commit_sent", 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.206244+0000", 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 0.0045564580000000002 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: }, 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: { 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "event": "done", 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "time": "2024-09-13T13:52:32.206293+0000", 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: "duration": 4.8581999999999997e-05 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: } 2024-09-13T13:52:49.223 INFO:teuthology.orchestra.run.smithi159.stdout: ] 2024-09-13T13:52:49.224 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:49.224 INFO:teuthology.orchestra.run.smithi027.stdout:osd.10: {} 2024-09-13T13:52:49.225 INFO:teuthology.orchestra.run.smithi027.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T13:52:49.226 INFO:teuthology.orchestra.run.smithi027.stdout:osd.11: {} 2024-09-13T13:52:49.226 INFO:teuthology.orchestra.run.smithi027.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T13:52:49.226 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:49.227 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:49.224+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:49.227 INFO:teuthology.orchestra.run.smithi027.stdout:osd.12: {} 2024-09-13T13:52:49.227 INFO:teuthology.orchestra.run.smithi027.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T13:52:49.227 INFO:teuthology.orchestra.run.smithi027.stdout:osd.13: {} 2024-09-13T13:52:49.228 INFO:teuthology.orchestra.run.smithi027.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T13:52:49.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.14: {} 2024-09-13T13:52:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T13:52:49.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:52:49.228+0000 7fcf3a59c640 0 -- 172.21.15.27:0/1682239941 >> v2:172.21.15.159:6805/4073091144 conn(0x7fcf3c220000 msgr2=0x7fcf3c2223f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T13:52:49.266 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is set 2024-09-13T13:52:49.269 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:49.268+0000 7f2284f10640 -1 osd.10 176 handle_osd_map injecting map crc failure 2024-09-13T13:52:49.325 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:49.325+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:49.370 INFO:teuthology.orchestra.run.smithi159.stdout:{ 2024-09-13T13:52:49.370 INFO:teuthology.orchestra.run.smithi159.stdout: "ops": [], 2024-09-13T13:52:49.370 INFO:teuthology.orchestra.run.smithi159.stdout: "num_ops": 0 2024-09-13T13:52:49.370 INFO:teuthology.orchestra.run.smithi159.stdout:} 2024-09-13T13:52:49.379 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:49.426 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:49.426+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:49.452 INFO:teuthology.orchestra.run.smithi027.stdout:osd.15: {} 2024-09-13T13:52:49.453 INFO:teuthology.orchestra.run.smithi027.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T13:52:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T13:52:49.456 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T13:52:49.483 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:49.483 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:49.492 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:49.527 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:49.526+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:49.594 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:49.594 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:49.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T13:52:49.628 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:49.627+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:49.711 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:49.711 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:49.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T13:52:49.728 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:49.727+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:49.827 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:49.827 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:49.829 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:49.829+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:49.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T13:52:49.892 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:49.891+0000 7f7c73c21640 -1 osd.14 176 handle_osd_map injecting map crc failure 2024-09-13T13:52:49.901 INFO:tasks.rados.rados.0.smithi159.stderr:0 errors. 2024-09-13T13:52:49.901 INFO:tasks.rados.rados.0.smithi159.stderr: 2024-09-13T13:52:49.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:52:49.944 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:49.944 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:49.946 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:49.945+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:49.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T13:52:50.037 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:52:50.036+0000 7ffafe2ef640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33209) UID: 0 2024-09-13T13:52:50.111 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.111 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T13:52:50.132 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:50.131+0000 7fc179787640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33591) UID: 0 2024-09-13T13:52:50.225 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:52:50.225 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":177,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:52:49.261047+0000","last_up_change":"2024-09-13T13:51:43.734671+0000","last_in_change":"2024-09-13T13:51:44.736036+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"reef","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T13:50:03.138857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9500000476837158,"score_stable":3.9500000476837158,"optimal_score":0.37999999523162842,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-09-13T13:50:04.433048+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":66,"snap_epoch":172,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.0099999904632568,"score_stable":3.0099999904632568,"optimal_score":0.81000000238418579,"raw_score_acting":2.440000057220459,"raw_score_stable":2.440000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":124,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":277461260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":277461260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":277461260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":277461260}]},"public_addr":"172.21.15.27:6812/277461260","cluster_addr":"172.21.15.27:6813/277461260","heartbeat_back_addr":"172.21.15.27:6815/277461260","heartbeat_front_addr":"172.21.15.27:6814/277461260","state":["exists"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:11.469586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:16.074960+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:14.772148+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:14.471039+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-13T13:50:50.576723+0000","dead_epoch":124},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:16.381078+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:09.863290+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:25.417884+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:35.308467+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:40.907494+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:11.716379+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:11.796248+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:12.588182+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:52:04.647603+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:47.173300+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:51.638027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":11,"to":15}]}],"pg_upmap_primaries":[{"pgid":"2.2","primary_osd":9}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:52:50.233 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:50.232+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:50.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:52:50.269 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.269 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T13:52:50.362 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:52:50.361+0000 7faeb9816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33216) UID: 0 2024-09-13T13:52:50.428 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.428 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.434 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:50.433+0000 7f7c81224640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33137) UID: 0 2024-09-13T13:52:50.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T13:52:50.535 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:52:50.534+0000 7fe105402640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33129) UID: 0 2024-09-13T13:52:50.544 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:52:50.545 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:52:50.558 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-13T13:52:50.397994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":521823646,"num_objects":343,"num_object_clones":136,"num_object_copies":686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":6,"num_read":5675,"num_read_kb":4446948,"num_write":2608,"num_write_kb":736974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":341569177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":103,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4874,"ondisk_log_size":4874,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":1245224,"kb_used_data":794104,"kb_used_omap":23,"kb_used_meta":450856,"kb_avail":1414332376,"statfs":{"total":1449551462400,"available":1448276353024,"internally_reserved":0,"allocated":813162496,"data_stored":992466911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23924,"internal_metadata":461677196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.644821"},"pg_stats":[{"pgid":"4.f","version":"172'311","reported_seq":1034,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.645870+0000","last_change":"2024-09-13T13:52:43.844746+0000","last_active":"2024-09-13T13:52:44.645870+0000","last_peered":"2024-09-13T13:52:44.645870+0000","last_clean":"2024-09-13T13:52:44.645870+0000","last_became_active":"2024-09-13T13:50:06.073131+0000","last_became_peered":"2024-09-13T13:50:06.073131+0000","last_unstale":"2024-09-13T13:52:44.645870+0000","last_undegraded":"2024-09-13T13:52:44.645870+0000","last_fullsized":"2024-09-13T13:52:44.645870+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'311","last_scrub_stamp":"2024-09-13T13:52:43.844641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:43.844641+0000","objects_scrubbed":14,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:50.338778+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.96058579899999996,"stat_sum":{"num_bytes":22490858,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":429,"num_read_kb":366489,"num_write":126,"num_write_kb":32087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.e","version":"172'326","reported_seq":1129,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.932035+0000","last_change":"2024-09-13T13:52:36.239738+0000","last_active":"2024-09-13T13:52:44.932035+0000","last_peered":"2024-09-13T13:52:44.932035+0000","last_clean":"2024-09-13T13:52:44.932035+0000","last_became_active":"2024-09-13T13:51:45.749931+0000","last_became_peered":"2024-09-13T13:51:45.749931+0000","last_unstale":"2024-09-13T13:52:44.932035+0000","last_undegraded":"2024-09-13T13:52:44.932035+0000","last_fullsized":"2024-09-13T13:52:44.932035+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'315","last_scrub_stamp":"2024-09-13T13:52:25.144492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:25.144492+0000","objects_scrubbed":10,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:32.687057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010685392,"stat_sum":{"num_bytes":19328653,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":532,"num_read_kb":443443,"num_write":126,"num_write_kb":33575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":14756273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"172'85","reported_seq":729,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.972511+0000","last_change":"2024-09-13T13:52:36.401314+0000","last_active":"2024-09-13T13:52:44.972511+0000","last_peered":"2024-09-13T13:52:44.972511+0000","last_clean":"2024-09-13T13:52:44.972511+0000","last_became_active":"2024-09-13T13:50:06.075524+0000","last_became_peered":"2024-09-13T13:50:06.075524+0000","last_unstale":"2024-09-13T13:52:44.972511+0000","last_undegraded":"2024-09-13T13:52:44.972511+0000","last_fullsized":"2024-09-13T13:52:44.972511+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'83","last_scrub_stamp":"2024-09-13T13:52:35.029316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:35.029316+0000","objects_scrubbed":4,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:39.471780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0090790520000000006,"stat_sum":{"num_bytes":6403889,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":119,"num_read_kb":109848,"num_write":27,"num_write_kb":6222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15],"acting":[12,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"172'376","reported_seq":1123,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:45.970841+0000","last_change":"2024-09-13T13:52:45.970841+0000","last_active":"2024-09-13T13:52:45.970841+0000","last_peered":"2024-09-13T13:52:45.970841+0000","last_clean":"2024-09-13T13:52:45.970841+0000","last_became_active":"2024-09-13T13:50:28.729220+0000","last_became_peered":"2024-09-13T13:50:28.729220+0000","last_unstale":"2024-09-13T13:52:45.970841+0000","last_undegraded":"2024-09-13T13:52:45.970841+0000","last_fullsized":"2024-09-13T13:52:45.970841+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'376","last_scrub_stamp":"2024-09-13T13:52:45.970736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:45.970736+0000","objects_scrubbed":18,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:13.157469+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0047818009999999996,"stat_sum":{"num_bytes":37976917,"num_objects":18,"num_object_clones":14,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":406,"num_read_kb":336511,"num_write":168,"num_write_kb":43350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11131276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,6],"acting":[12,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"172'388","reported_seq":1081,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.694826+0000","last_change":"2024-09-13T13:52:44.820577+0000","last_active":"2024-09-13T13:52:47.694826+0000","last_peered":"2024-09-13T13:52:47.694826+0000","last_clean":"2024-09-13T13:52:47.694826+0000","last_became_active":"2024-09-13T13:50:06.073188+0000","last_became_peered":"2024-09-13T13:50:06.073188+0000","last_unstale":"2024-09-13T13:52:47.694826+0000","last_undegraded":"2024-09-13T13:52:47.694826+0000","last_fullsized":"2024-09-13T13:52:47.694826+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'388","last_scrub_stamp":"2024-09-13T13:52:44.820478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:44.820478+0000","objects_scrubbed":18,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:58.675662+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0072376990000000002,"stat_sum":{"num_bytes":39274374,"num_objects":18,"num_object_clones":13,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":360,"num_read_kb":298301,"num_write":192,"num_write_kb":53199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2],"acting":[14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"172'324","reported_seq":1120,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.725822+0000","last_change":"2024-09-13T13:52:46.235781+0000","last_active":"2024-09-13T13:52:47.725822+0000","last_peered":"2024-09-13T13:52:47.725822+0000","last_clean":"2024-09-13T13:52:47.725822+0000","last_became_active":"2024-09-13T13:51:02.821659+0000","last_became_peered":"2024-09-13T13:51:02.821659+0000","last_unstale":"2024-09-13T13:52:47.725822+0000","last_undegraded":"2024-09-13T13:52:47.725822+0000","last_fullsized":"2024-09-13T13:52:47.725822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'324","last_scrub_stamp":"2024-09-13T13:52:46.235719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:46.235719+0000","objects_scrubbed":17,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:03.710096+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0062566000000000002,"stat_sum":{"num_bytes":36077090,"num_objects":17,"num_object_clones":13,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":407,"num_read_kb":386458,"num_write":131,"num_write_kb":36214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":34232380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"172'76","reported_seq":690,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:46.248750+0000","last_change":"2024-09-13T13:52:42.636182+0000","last_active":"2024-09-13T13:52:46.248750+0000","last_peered":"2024-09-13T13:52:46.248750+0000","last_clean":"2024-09-13T13:52:46.248750+0000","last_became_active":"2024-09-13T13:50:12.197810+0000","last_became_peered":"2024-09-13T13:50:12.197810+0000","last_unstale":"2024-09-13T13:52:46.248750+0000","last_undegraded":"2024-09-13T13:52:46.248750+0000","last_fullsized":"2024-09-13T13:52:46.248750+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'76","last_scrub_stamp":"2024-09-13T13:52:42.636116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:42.636116+0000","objects_scrubbed":3,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:58.815163+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0021530099999999999,"stat_sum":{"num_bytes":4358231,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":87,"num_read_kb":74224,"num_write":27,"num_write_kb":6010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,5],"acting":[15,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"172'217","reported_seq":840,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:43.758857+0000","last_change":"2024-09-13T13:52:36.221809+0000","last_active":"2024-09-13T13:52:43.758857+0000","last_peered":"2024-09-13T13:52:43.758857+0000","last_clean":"2024-09-13T13:52:43.758857+0000","last_became_active":"2024-09-13T13:50:06.075916+0000","last_became_peered":"2024-09-13T13:50:06.075916+0000","last_unstale":"2024-09-13T13:52:43.758857+0000","last_undegraded":"2024-09-13T13:52:43.758857+0000","last_fullsized":"2024-09-13T13:52:43.758857+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'120","last_scrub_stamp":"2024-09-13T13:51:33.575276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:33.575276+0000","objects_scrubbed":6,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:57.871038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029113530000000002,"stat_sum":{"num_bytes":23473427,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":167,"num_read_kb":130383,"num_write":90,"num_write_kb":24670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11],"acting":[3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"172'329","reported_seq":938,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.670425+0000","last_change":"2024-09-13T13:52:36.219841+0000","last_active":"2024-09-13T13:52:44.670425+0000","last_peered":"2024-09-13T13:52:44.670425+0000","last_clean":"2024-09-13T13:52:44.670425+0000","last_became_active":"2024-09-13T13:51:05.828626+0000","last_became_peered":"2024-09-13T13:51:05.828626+0000","last_unstale":"2024-09-13T13:52:44.670425+0000","last_undegraded":"2024-09-13T13:52:44.670425+0000","last_fullsized":"2024-09-13T13:52:44.670425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'321","last_scrub_stamp":"2024-09-13T13:52:31.642516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:31.642516+0000","objects_scrubbed":16,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:40.541785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0054189950000000002,"stat_sum":{"num_bytes":35979810,"num_objects":16,"num_object_clones":12,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":378,"num_read_kb":309088,"num_write":146,"num_write_kb":42994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":40416780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"27'112","reported_seq":413,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.670486+0000","last_change":"2024-09-13T13:51:09.467323+0000","last_active":"2024-09-13T13:52:44.670486+0000","last_peered":"2024-09-13T13:52:44.670486+0000","last_clean":"2024-09-13T13:52:44.670486+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:52:44.670486+0000","last_undegraded":"2024-09-13T13:52:44.670486+0000","last_fullsized":"2024-09-13T13:52:44.670486+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:51:09.467249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:51:09.467249+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:44.700440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":328,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:46.247448+0000","last_change":"2024-09-13T13:52:34.840060+0000","last_active":"2024-09-13T13:52:46.247448+0000","last_peered":"2024-09-13T13:52:46.247448+0000","last_clean":"2024-09-13T13:52:46.247448+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:52:46.247448+0000","last_undegraded":"2024-09-13T13:52:46.247448+0000","last_fullsized":"2024-09-13T13:52:46.247448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:52:34.839984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:52:34.839984+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:50.052124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":658,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.718041+0000","last_change":"2024-09-13T13:52:41.844392+0000","last_active":"2024-09-13T13:52:47.718041+0000","last_peered":"2024-09-13T13:52:47.718041+0000","last_clean":"2024-09-13T13:52:47.718041+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:52:47.718041+0000","last_undegraded":"2024-09-13T13:52:47.718041+0000","last_fullsized":"2024-09-13T13:52:47.718041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:52:41.844309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:52:41.844309+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:04.731163+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.6","version":"172'189","reported_seq":945,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.719910+0000","last_change":"2024-09-13T13:52:36.224017+0000","last_active":"2024-09-13T13:52:47.719910+0000","last_peered":"2024-09-13T13:52:47.719910+0000","last_clean":"2024-09-13T13:52:47.719910+0000","last_became_active":"2024-09-13T13:51:00.088253+0000","last_became_peered":"2024-09-13T13:51:00.088253+0000","last_unstale":"2024-09-13T13:52:47.719910+0000","last_undegraded":"2024-09-13T13:52:47.719910+0000","last_fullsized":"2024-09-13T13:52:47.719910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'182","last_scrub_stamp":"2024-09-13T13:52:33.009715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:33.009715+0000","objects_scrubbed":8,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:01.294187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0056967179999999999,"stat_sum":{"num_bytes":15236419,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":304,"num_read_kb":275712,"num_write":82,"num_write_kb":22085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14889074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":323,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.234448+0000","last_change":"2024-09-13T13:51:26.177165+0000","last_active":"2024-09-13T13:52:47.234448+0000","last_peered":"2024-09-13T13:52:47.234448+0000","last_clean":"2024-09-13T13:52:47.234448+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:52:47.234448+0000","last_undegraded":"2024-09-13T13:52:47.234448+0000","last_fullsized":"2024-09-13T13:52:47.234448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:26.177104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:26.177104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":597,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.645838+0000","last_change":"2024-09-13T13:51:49.843053+0000","last_active":"2024-09-13T13:52:44.645838+0000","last_peered":"2024-09-13T13:52:44.645838+0000","last_clean":"2024-09-13T13:52:44.645838+0000","last_became_active":"2024-09-13T13:51:45.952794+0000","last_became_peered":"2024-09-13T13:51:45.952794+0000","last_unstale":"2024-09-13T13:52:44.645838+0000","last_undegraded":"2024-09-13T13:52:44.645838+0000","last_fullsized":"2024-09-13T13:52:44.645838+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'152","last_scrub_stamp":"2024-09-13T13:51:49.842939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:49.842939+0000","objects_scrubbed":34,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:00.129584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":135,"num_read_kb":8629,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30181163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"172'280","reported_seq":988,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.731664+0000","last_change":"2024-09-13T13:52:36.396767+0000","last_active":"2024-09-13T13:52:47.731664+0000","last_peered":"2024-09-13T13:52:47.731664+0000","last_clean":"2024-09-13T13:52:47.731664+0000","last_became_active":"2024-09-13T13:50:29.051980+0000","last_became_peered":"2024-09-13T13:50:29.051980+0000","last_unstale":"2024-09-13T13:52:47.731664+0000","last_undegraded":"2024-09-13T13:52:47.731664+0000","last_fullsized":"2024-09-13T13:52:47.731664+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'274","last_scrub_stamp":"2024-09-13T13:52:35.354057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:35.354057+0000","objects_scrubbed":13,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:41.871172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.004334727,"stat_sum":{"num_bytes":26574462,"num_objects":13,"num_object_clones":10,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":258,"num_read_kb":229672,"num_write":123,"num_write_kb":32905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9395884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:46.247432+0000","last_change":"2024-09-13T13:52:24.890523+0000","last_active":"2024-09-13T13:52:46.247432+0000","last_peered":"2024-09-13T13:52:46.247432+0000","last_clean":"2024-09-13T13:52:46.247432+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:52:46.247432+0000","last_undegraded":"2024-09-13T13:52:46.247432+0000","last_fullsized":"2024-09-13T13:52:46.247432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:52:24.890445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:52:24.890445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:35.366303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":368,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:43.758816+0000","last_change":"2024-09-13T13:52:43.757338+0000","last_active":"2024-09-13T13:52:43.758816+0000","last_peered":"2024-09-13T13:52:43.758816+0000","last_clean":"2024-09-13T13:52:43.758816+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:52:43.758816+0000","last_undegraded":"2024-09-13T13:52:43.758816+0000","last_fullsized":"2024-09-13T13:52:43.758816+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:52:43.757234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:52:43.757234+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:54:04.639983+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":128,"num_read_kb":4141,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"172'265","reported_seq":599,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.670421+0000","last_change":"2024-09-13T13:52:36.223641+0000","last_active":"2024-09-13T13:52:44.670421+0000","last_peered":"2024-09-13T13:52:44.670421+0000","last_clean":"2024-09-13T13:52:44.670421+0000","last_became_active":"2024-09-13T13:51:02.819431+0000","last_became_peered":"2024-09-13T13:51:02.819431+0000","last_unstale":"2024-09-13T13:52:44.670421+0000","last_undegraded":"2024-09-13T13:52:44.670421+0000","last_fullsized":"2024-09-13T13:52:44.670421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'136","last_scrub_stamp":"2024-09-13T13:51:34.295925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:34.295925+0000","objects_scrubbed":9,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:01.466040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0091876019999999996,"stat_sum":{"num_bytes":26582521,"num_objects":15,"num_object_clones":12,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":245,"num_read_kb":228096,"num_write":124,"num_write_kb":35396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":27682344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":10,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":343,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:45.603124+0000","last_change":"2024-09-13T13:52:25.309210+0000","last_active":"2024-09-13T13:52:45.603124+0000","last_peered":"2024-09-13T13:52:45.603124+0000","last_clean":"2024-09-13T13:52:45.603124+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:52:45.603124+0000","last_undegraded":"2024-09-13T13:52:45.603124+0000","last_fullsized":"2024-09-13T13:52:45.603124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:52:25.309134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:52:25.309134+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:50.384231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":473,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.931331+0000","last_change":"2024-09-13T13:51:23.029600+0000","last_active":"2024-09-13T13:52:44.931331+0000","last_peered":"2024-09-13T13:52:44.931331+0000","last_clean":"2024-09-13T13:52:44.931331+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:52:44.931331+0000","last_undegraded":"2024-09-13T13:52:44.931331+0000","last_fullsized":"2024-09-13T13:52:44.931331+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'125","last_scrub_stamp":"2024-09-13T13:51:23.029522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:51:23.029522+0000","objects_scrubbed":20,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:45.983988+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":130,"num_read_kb":6618,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"172'277","reported_seq":818,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.670672+0000","last_change":"2024-09-13T13:52:36.218085+0000","last_active":"2024-09-13T13:52:44.670672+0000","last_peered":"2024-09-13T13:52:44.670672+0000","last_clean":"2024-09-13T13:52:44.670672+0000","last_became_active":"2024-09-13T13:51:06.832059+0000","last_became_peered":"2024-09-13T13:51:06.832059+0000","last_unstale":"2024-09-13T13:52:44.670672+0000","last_undegraded":"2024-09-13T13:52:44.670672+0000","last_fullsized":"2024-09-13T13:52:44.670672+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'262","last_scrub_stamp":"2024-09-13T13:52:25.572449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:25.572449+0000","objects_scrubbed":10,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:55.007361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003699117,"stat_sum":{"num_bytes":25796292,"num_objects":11,"num_object_clones":7,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":389,"num_read_kb":382236,"num_write":127,"num_write_kb":34960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":33977752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":319,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.219487+0000","last_change":"2024-09-13T13:51:13.190624+0000","last_active":"2024-09-13T13:52:47.219487+0000","last_peered":"2024-09-13T13:52:47.219487+0000","last_clean":"2024-09-13T13:52:47.219487+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:52:47.219487+0000","last_undegraded":"2024-09-13T13:52:47.219487+0000","last_fullsized":"2024-09-13T13:52:47.219487+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:13.190561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.190561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"172'69","reported_seq":706,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:45.302143+0000","last_change":"2024-09-13T13:52:45.302143+0000","last_active":"2024-09-13T13:52:45.302143+0000","last_peered":"2024-09-13T13:52:45.302143+0000","last_clean":"2024-09-13T13:52:45.302143+0000","last_became_active":"2024-09-13T13:50:06.075820+0000","last_became_peered":"2024-09-13T13:50:06.075820+0000","last_unstale":"2024-09-13T13:52:45.302143+0000","last_undegraded":"2024-09-13T13:52:45.302143+0000","last_fullsized":"2024-09-13T13:52:45.302143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'69","last_scrub_stamp":"2024-09-13T13:52:45.302060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:52:45.302060+0000","objects_scrubbed":4,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:52.772043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001927004,"stat_sum":{"num_bytes":6525777,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":101,"num_read_kb":95304,"num_write":29,"num_write_kb":7898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":331,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.705611+0000","last_change":"2024-09-13T13:52:13.768640+0000","last_active":"2024-09-13T13:52:47.705611+0000","last_peered":"2024-09-13T13:52:47.705611+0000","last_clean":"2024-09-13T13:52:47.705611+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:52:47.705611+0000","last_undegraded":"2024-09-13T13:52:47.705611+0000","last_fullsized":"2024-09-13T13:52:47.705611+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:52:13.768578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:52:13.768578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:30.921390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"172'101","reported_seq":751,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.192702+0000","last_change":"2024-09-13T13:52:36.213674+0000","last_active":"2024-09-13T13:52:44.192702+0000","last_peered":"2024-09-13T13:52:44.192702+0000","last_clean":"2024-09-13T13:52:44.192702+0000","last_became_active":"2024-09-13T13:51:45.749549+0000","last_became_peered":"2024-09-13T13:51:45.749549+0000","last_unstale":"2024-09-13T13:52:44.192702+0000","last_undegraded":"2024-09-13T13:52:44.192702+0000","last_fullsized":"2024-09-13T13:52:44.192702+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'59","last_scrub_stamp":"2024-09-13T13:51:22.810355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.810355+0000","objects_scrubbed":3,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:47.240730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022597509999999999,"stat_sum":{"num_bytes":9666779,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":118,"num_read_kb":101582,"num_write":37,"num_write_kb":8132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3724338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":323,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:45.603124+0000","last_change":"2024-09-13T13:52:24.259770+0000","last_active":"2024-09-13T13:52:45.603124+0000","last_peered":"2024-09-13T13:52:45.603124+0000","last_clean":"2024-09-13T13:52:45.603124+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:52:45.603124+0000","last_undegraded":"2024-09-13T13:52:45.603124+0000","last_fullsized":"2024-09-13T13:52:45.603124+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:52:24.259648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:52:24.259648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:53:37.178423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"172'482","reported_seq":1361,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-13T13:52:44.932026+0000","last_change":"2024-09-13T13:52:36.242272+0000","last_active":"2024-09-13T13:52:44.932026+0000","last_peered":"2024-09-13T13:52:44.932026+0000","last_clean":"2024-09-13T13:52:44.932026+0000","last_became_active":"2024-09-13T13:50:54.737668+0000","last_became_peered":"2024-09-13T13:50:54.737668+0000","last_unstale":"2024-09-13T13:52:44.932026+0000","last_undegraded":"2024-09-13T13:52:44.932026+0000","last_fullsized":"2024-09-13T13:52:44.932026+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'211","last_scrub_stamp":"2024-09-13T13:51:13.786427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:05.059988+0000","last_clean_scrub_stamp":"2024-09-13T13:51:13.786427+0000","objects_scrubbed":13,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:45.254897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013253876,"stat_sum":{"num_bytes":45857062,"num_objects":23,"num_object_clones":17,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":741,"num_read_kb":652558,"num_write":187,"num_write_kb":50557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":29261751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":16,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":323,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-09-13T13:52:47.234554+0000","last_change":"2024-09-13T13:51:22.235733+0000","last_active":"2024-09-13T13:52:47.234554+0000","last_peered":"2024-09-13T13:52:47.234554+0000","last_clean":"2024-09-13T13:52:47.234554+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:52:47.234554+0000","last_undegraded":"2024-09-13T13:52:47.234554+0000","last_fullsized":"2024-09-13T13:52:47.234554+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:22.235634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:51:22.235634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:52:51.179878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":381602561,"num_objects":186,"num_object_clones":136,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":6,"num_read":5041,"num_read_kb":4419905,"num_write":1742,"num_write_kb":470254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":219467852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":103,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":551981056,"data_stored":748640674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4095,"ondisk_log_size":4095,"up":32,"acting":32,"num_store_stats":15},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":548,"num_read_kb":26970,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":121510919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330240000,"data_stored":329628729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39104,"kb_used_data":7920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332736,"statfs":{"total":96636764160,"available":96596721664,"internally_reserved":0,"allocated":8110080,"data_stored":11807303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247847,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":106104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":108650496,"data_stored":114982085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49432,"kb_used_data":18248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322408,"statfs":{"total":96636764160,"available":96586145792,"internally_reserved":0,"allocated":18685952,"data_stored":20692110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280550,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123192,"kb_used_data":92008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94248648,"statfs":{"total":96636764160,"available":96510615552,"internally_reserved":0,"allocated":94216192,"data_stored":126143135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280550,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117916,"kb_used_data":86732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94253924,"statfs":{"total":96636764160,"available":96516018176,"internally_reserved":0,"allocated":88813568,"data_stored":90028471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280550,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54096,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317744,"statfs":{"total":96636764160,"available":96581369856,"internally_reserved":0,"allocated":27721728,"data_stored":46511177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280551,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170848,"kb_used_data":135504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94200992,"statfs":{"total":96636764160,"available":96461815808,"internally_reserved":0,"allocated":138756096,"data_stored":189870666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280550,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":492147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280550,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79052,"kb_used_data":52028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292788,"statfs":{"total":96636764160,"available":96555814912,"internally_reserved":0,"allocated":53276672,"data_stored":55990574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56044,"kb_used_data":29020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94315796,"statfs":{"total":96636764160,"available":96579375104,"internally_reserved":0,"allocated":29716480,"data_stored":39498663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313255,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94196,"kb_used_data":63012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277644,"statfs":{"total":96636764160,"available":96540307456,"internally_reserved":0,"allocated":64524288,"data_stored":74549504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":741376,"data_stored":507401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49980,"kb_used_data":18796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321860,"statfs":{"total":96636764160,"available":96585584640,"internally_reserved":0,"allocated":19247104,"data_stored":30101986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73892,"kb_used_data":42708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297948,"statfs":{"total":96636764160,"available":96561098752,"internally_reserved":0,"allocated":43732992,"data_stored":44678342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,3,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144708,"kb_used_data":113524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94227132,"statfs":{"total":96636764160,"available":96488583168,"internally_reserved":0,"allocated":116248576,"data_stored":146613347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30228480,"data_stored":30181163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":79101952,"data_stored":78964016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":18655977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18538496,"data_stored":29598076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93495296,"data_stored":125644061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12988416,"data_stored":14569669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16887808,"data_stored":25492208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26976256,"data_stored":46008398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":137420800,"data_stored":188778140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11325440,"data_stored":12567632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":86380544,"data_stored":117053438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":23437312,"data_stored":26434868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17960960,"data_stored":20186946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":28983296,"data_stored":38998133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":28090368,"data_stored":38430857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":28798976,"data_stored":35509031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":6803456,"data_stored":10713240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:52:50.560 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T13:52:50.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T13:52:50.587 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.588 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T13:52:50.636 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:50.634+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:50.737 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:50.736+0000 7f228e50a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33134) UID: 0 2024-09-13T13:52:50.753 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.753 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T13:52:50.838 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:50.836+0000 7f9b17b52640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33217) UID: 0 2024-09-13T13:52:50.870 INFO:teuthology.orchestra.run.smithi027.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.870 INFO:teuthology.orchestra.run.smithi027.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:50.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T13:52:50.938 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:52:50.938+0000 7f8f64260640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33132) UID: 0 2024-09-13T13:52:50.992 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:50.992 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T13:52:51.045 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:52:51.044+0000 7f5879d1b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33148) UID: 0 2024-09-13T13:52:51.104 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.105 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.113 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T13:52:51.140 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:52:51.139+0000 7f26c65dd640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33586) UID: 0 2024-09-13T13:52:51.240 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:52:51.239+0000 7f500b381640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33142) UID: 0 2024-09-13T13:52:51.256 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.256 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T13:52:51.272 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'unique_pool_0' removed 2024-09-13T13:52:51.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:51.288 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T13:52:51.297 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T13:52:51.409 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.409 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T13:52:51.521 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.521 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.530 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T13:52:51.633 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.633 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.642 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T13:52:51.745 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.746 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.754 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T13:52:51.856 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.856 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T13:52:51.967 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:51.967 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:51.976 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T13:52:52.082 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.082 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.090 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T13:52:52.192 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.193 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.201 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T13:52:52.284 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:52.285 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:52:52.285+0000 7fe859f53640 -1 osd.7 178 handle_osd_map injecting map crc failure 2024-09-13T13:52:52.305 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.305 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.314 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T13:52:52.436 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.436 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T13:52:52.551 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.551 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T13:52:52.618 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:52:52.617+0000 7f1a506ea640 -1 osd.13 178 handle_osd_map injecting map crc failure 2024-09-13T13:52:52.618 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:52:52.617+0000 7f9b0c553640 -1 osd.12 178 handle_osd_map injecting map crc failure 2024-09-13T13:52:52.664 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.664 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.673 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T13:52:52.777 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.777 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T13:52:52.888 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:52.888 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:52.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T13:52:52.999 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.000 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.009 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T13:52:53.111 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.111 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.120 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T13:52:53.221 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.222 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T13:52:53.334 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.334 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T13:52:53.446 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.446 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T13:52:53.558 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.558 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T13:52:53.671 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.671 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.680 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T13:52:53.795 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.795 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.804 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T13:52:53.907 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:53.907 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:53.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T13:52:54.020 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.020 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.029 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T13:52:54.131 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.131 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.140 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T13:52:54.244 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.244 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.252 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T13:52:54.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T13:52:54.356 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.356 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.365 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T13:52:54.470 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.470 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.479 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T13:52:54.582 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.582 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.591 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T13:52:54.696 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.696 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T13:52:54.809 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.809 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.817 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T13:52:54.919 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:54.919 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:54.927 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T13:52:55.030 INFO:teuthology.orchestra.run.smithi159.stdout:ERROR: (22) Invalid argument 2024-09-13T13:52:55.031 INFO:teuthology.orchestra.run.smithi159.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T13:52:55.297 INFO:teuthology.orchestra.run.smithi027.stderr:noscrub is unset 2024-09-13T13:52:55.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T13:52:55.784 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:52:55.783+0000 7f002fbdf640 -1 osd.15 179 handle_osd_map injecting map crc failure 2024-09-13T13:52:56.304 INFO:teuthology.orchestra.run.smithi027.stderr:nodeep-scrub is unset 2024-09-13T13:52:56.304 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:52:56.303+0000 7fc16c184640 -1 osd.11 180 handle_osd_map injecting map crc failure 2024-09-13T13:52:56.452 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:52:56.451+0000 7f7c71c1d640 -1 osd.14 179 handle_osd_map injecting map crc failure 2024-09-13T13:52:56.455 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:52:56.454+0000 7f2282f0c640 -1 osd.10 180 handle_osd_map injecting map crc failure 2024-09-13T13:53:01.207 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:53:01.206+0000 7faeac214640 -1 osd.8 181 handle_osd_map injecting map crc failure 2024-09-13T13:53:21.782 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:53:21.780+0000 7f2284f10640 -1 osd.10 183 handle_osd_map injecting map crc failure 2024-09-13T13:53:24.934 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:53:24.932+0000 7f3c35934640 -1 osd.5 184 handle_osd_map injecting map crc failure 2024-09-13T13:53:32.816 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:53:32.814+0000 7fc16c184640 -1 osd.11 186 handle_osd_map injecting map crc failure 2024-09-13T13:53:36.829 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:53:36.829+0000 7f586c719640 -1 osd.9 188 handle_osd_map injecting map crc failure 2024-09-13T13:53:37.829 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:53:37.827+0000 7f9b0e557640 -1 osd.12 188 handle_osd_map injecting map crc failure 2024-09-13T13:53:37.831 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:53:37.830+0000 7f3c37938640 -1 osd.5 188 handle_osd_map injecting map crc failure 2024-09-13T13:53:38.164 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:53:38.161+0000 7fe857f4f640 -1 osd.7 188 handle_osd_map injecting map crc failure 2024-09-13T13:53:38.194 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:53:38.194+0000 7f4ffbd7a640 -1 osd.1 188 handle_osd_map injecting map crc failure 2024-09-13T13:53:38.959 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:53:38.957+0000 7faeaa210640 -1 osd.8 190 handle_osd_map injecting map crc failure 2024-09-13T13:53:55.498 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:53:55.497+0000 7f9b0c553640 -1 osd.12 193 handle_osd_map injecting map crc failure 2024-09-13T13:53:55.563 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:53:55.562+0000 7f002dbdb640 -1 osd.15 193 handle_osd_map injecting map crc failure 2024-09-13T13:53:58.898 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:53:58.897+0000 7f7c73c21640 -1 osd.14 194 handle_osd_map injecting map crc failure 2024-09-13T13:53:59.201 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:53:59.200+0000 7f26bafde640 -1 osd.3 194 handle_osd_map injecting map crc failure 2024-09-13T13:54:23.403 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:54:23.400+0000 7f7c71c1d640 -1 osd.14 196 handle_osd_map injecting map crc failure 2024-09-13T13:54:28.783 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:54:28.782+0000 7f1a506ea640 -1 osd.13 197 handle_osd_map injecting map crc failure 2024-09-13T13:54:38.990 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:54:38.988+0000 7fc16c184640 -1 osd.11 198 handle_osd_map injecting map crc failure 2024-09-13T13:54:38.991 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:54:38.990+0000 7f3c37938640 -1 osd.5 198 handle_osd_map injecting map crc failure 2024-09-13T13:54:39.991 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:54:39.989+0000 7faeac214640 -1 osd.8 199 handle_osd_map injecting map crc failure 2024-09-13T13:54:45.185 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:54:45.183+0000 7f1a506ea640 -1 osd.13 200 handle_osd_map injecting map crc failure 2024-09-13T13:54:45.185 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:54:45.184+0000 7f7c71c1d640 -1 osd.14 200 handle_osd_map injecting map crc failure 2024-09-13T13:54:50.570 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:54:50.568+0000 7f8f58c61640 -1 osd.6 202 handle_osd_map injecting map crc failure 2024-09-13T13:54:50.570 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:54:50.569+0000 7f8f58c61640 -1 osd.6 202 handle_osd_map injecting map crc failure 2024-09-13T13:54:50.964 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:54:50.963+0000 7f26bcfe2640 -1 osd.3 202 handle_osd_map injecting map crc failure 2024-09-13T13:54:51.208 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:54:51.206+0000 7f4ffbd7a640 -1 osd.1 202 handle_osd_map injecting map crc failure 2024-09-13T13:54:51.558 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:54:51.557+0000 7ffaeece8640 -1 osd.0 202 handle_osd_map injecting map crc failure 2024-09-13T13:54:52.403 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:54:52.402+0000 7fe0f9e03640 -1 osd.2 204 handle_osd_map injecting map crc failure 2024-09-13T13:54:53.334 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:54:53.334+0000 7f26bafde640 -1 osd.3 204 handle_osd_map injecting map crc failure 2024-09-13T13:54:53.465 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:54:53.464+0000 7f8f58c61640 -1 osd.6 204 handle_osd_map injecting map crc failure 2024-09-13T13:54:56.062 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:54:56.061+0000 7f586c719640 -1 osd.9 205 handle_osd_map injecting map crc failure 2024-09-13T13:54:56.062 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:54:56.061+0000 7f7c73c21640 -1 osd.14 205 handle_osd_map injecting map crc failure 2024-09-13T13:54:56.835 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:54:56.835+0000 7f1a506ea640 -1 osd.13 205 handle_osd_map injecting map crc failure 2024-09-13T13:54:59.118 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:54:59.117+0000 7f9b0c553640 -1 osd.12 206 handle_osd_map injecting map crc failure 2024-09-13T13:54:59.266 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:54:59.265+0000 7f8f58c61640 -1 osd.6 206 handle_osd_map injecting map crc failure 2024-09-13T13:55:01.106 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:55:01.104+0000 7faeac214640 -1 osd.8 208 handle_osd_map injecting map crc failure 2024-09-13T13:55:11.141 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-13T13:55:11.142 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T13:55:11.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T13:55:11.146 INFO:tasks.ceph.osd.4:Started 2024-09-13T13:55:11.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:config diff 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:config diff get 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:config get 2024-09-13T13:55:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:config help [] 2024-09-13T13:55:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:config set ... 2024-09-13T13:55:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:admin_socket: invalid command 2024-09-13T13:55:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:55:11.262 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-13T13:55:12.582 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:12.582+0000 7f564def1640 -1 Falling back to public interface 2024-09-13T13:55:16.106 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.106+0000 7f564def1640 -1 osd.4 123 log_to_monitors true 2024-09-13T13:55:16.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:no valid command found; 10 closest matches: 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:0 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:1 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:abort 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:assert 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs debug_inject_read_zeros 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs files list 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:bluefs stats 2024-09-13T13:55:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator dump block 2024-09-13T13:55:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:bluestore allocator fragmentation block 2024-09-13T13:55:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:admin_socket: invalid command 2024-09-13T13:55:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T13:55:16.381 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-13T13:55:16.802 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.801+0000 7f563d892640 -1 osd.4 123 handle_osd_map injecting map crc failure 2024-09-13T13:55:16.812 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 124, got 0 bytes 2024-09-13T13:55:16.812 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 125, got 0 bytes 2024-09-13T13:55:16.812 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 126, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 127, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 128, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 129, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 130, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 131, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 132, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 133, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 134, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 135, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 136, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 137, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.804+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 138, got 0 bytes 2024-09-13T13:55:16.813 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 139, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 140, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 141, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 142, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 143, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 144, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 145, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 146, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 147, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 148, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 149, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 150, got 0 bytes 2024-09-13T13:55:16.814 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 151, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 152, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 153, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 154, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 155, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562c45a640 -1 osd.4 157 failed to load OSD map for epoch 156, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 124, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 124, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 125, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 126, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 127, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 128, got 0 bytes 2024-09-13T13:55:16.815 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 129, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 130, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 131, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 132, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 133, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 125, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 126, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 127, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 128, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 129, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 130, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 131, got 0 bytes 2024-09-13T13:55:16.816 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 132, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 134, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 135, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 136, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 137, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 138, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 139, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 140, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 141, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 142, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.805+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 143, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 144, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 145, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 146, got 0 bytes 2024-09-13T13:55:16.817 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 147, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 148, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 149, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 150, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 151, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 152, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 153, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 154, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 155, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f562bc59640 -1 osd.4 157 failed to load OSD map for epoch 156, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 133, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 134, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 135, got 0 bytes 2024-09-13T13:55:16.818 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 136, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 137, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 138, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 139, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 140, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 141, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 142, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 143, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 144, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 145, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 146, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 147, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 148, got 0 bytes 2024-09-13T13:55:16.819 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 149, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 150, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 151, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 152, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 153, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 154, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 155, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.806+0000 7f5629454640 -1 osd.4 157 failed to load OSD map for epoch 156, got 0 bytes 2024-09-13T13:55:16.820 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:55:16.811+0000 7f26bafde640 -1 osd.3 209 handle_osd_map injecting map crc failure 2024-09-13T13:55:16.826 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.826+0000 7f563d892640 -1 osd.4 197 handle_osd_map injecting map crc failure 2024-09-13T13:55:16.848 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:16.848+0000 7f5641c9f640 -1 osd.4 210 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:55:16.977 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:55:16.977+0000 7fc16a180640 -1 osd.11 209 handle_osd_map injecting map crc failure 2024-09-13T13:55:17.869 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:55:17.868+0000 7ffaeece8640 -1 osd.0 210 handle_osd_map injecting map crc failure 2024-09-13T13:55:18.839 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:55:18.838+0000 7f7c71c1d640 -1 osd.14 211 handle_osd_map injecting map crc failure 2024-09-13T13:55:18.839 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:18.838+0000 7fe0f9e03640 -1 osd.2 211 handle_osd_map injecting map crc failure 2024-09-13T13:55:18.912 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:55:18.910+0000 7f586a715640 -1 osd.9 211 handle_osd_map injecting map crc failure 2024-09-13T13:55:18.912 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:55:18.911+0000 7f586a715640 -1 osd.9 211 handle_osd_map injecting map crc failure 2024-09-13T13:55:21.382 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T13:55:21.503 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:55:21.503 INFO:teuthology.orchestra.run.smithi027.stdout: "ops": [], 2024-09-13T13:55:21.503 INFO:teuthology.orchestra.run.smithi027.stdout: "num_ops": 0 2024-09-13T13:55:21.503 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:55:21.512 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T13:55:21.512 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T13:55:21.513 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-13T13:55:21.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T13:55:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T13:55:24.664 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:55:24.663+0000 7f7c71c1d640 -1 osd.14 213 handle_osd_map injecting map crc failure 2024-09-13T13:55:27.838 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T13:55:27.838 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-13T13:55:27.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-09-13T13:55:28.164 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-13T13:55:28.163+0000 7fe36cfd9640 -1 mon.a@0(leader).osd e214 definitely_dead 0 2024-09-13T13:55:28.834 INFO:teuthology.orchestra.run.smithi027.stderr:marked in osd.4. 2024-09-13T13:55:28.848 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-13T13:55:28.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T13:55:28.976 INFO:teuthology.orchestra.run.smithi027.stdout:mon.a: {} 2024-09-13T13:55:28.976 INFO:teuthology.orchestra.run.smithi027.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:55:28.992 INFO:teuthology.orchestra.run.smithi027.stdout:mon.b: {} 2024-09-13T13:55:28.992 INFO:teuthology.orchestra.run.smithi027.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:55:29.004 INFO:teuthology.orchestra.run.smithi027.stdout:mon.c: {} 2024-09-13T13:55:29.005 INFO:teuthology.orchestra.run.smithi027.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T13:55:29.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T13:55:29.839 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T13:55:29.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T13:55:30.472 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:55:30.470+0000 7f002dbdb640 -1 osd.15 214 handle_osd_map injecting map crc failure 2024-09-13T13:55:30.844 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T13:55:30.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T13:55:31.190 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:55:31.190+0000 7f3c35934640 -1 osd.5 216 handle_osd_map injecting map crc failure 2024-09-13T13:55:31.313 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:31.312+0000 7fe0f9e03640 -1 osd.2 216 handle_osd_map injecting map crc failure 2024-09-13T13:55:31.314 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:55:31.313+0000 7f7c71c1d640 -1 osd.14 216 handle_osd_map injecting map crc failure 2024-09-13T13:55:31.314 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:55:31.312+0000 7fe857f4f640 -1 osd.7 216 handle_osd_map injecting map crc failure 2024-09-13T13:55:31.624 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:55:31.622+0000 7f002dbdb640 -1 osd.15 216 handle_osd_map injecting map crc failure 2024-09-13T13:55:31.845 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T13:55:31.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-13T13:55:32.851 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-13T13:55:32.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T13:55:33.614 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:55:33.613+0000 7f7c71c1d640 -1 osd.14 217 handle_osd_map injecting map crc failure 2024-09-13T13:55:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T13:55:33.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-13T13:55:33.979 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:55:33.977+0000 7faeaa210640 -1 osd.8 219 handle_osd_map injecting map crc failure 2024-09-13T13:55:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-13T13:55:34.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T13:55:35.872 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T13:55:35.875 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:35.874+0000 7fe0fbe07640 -1 osd.2 221 handle_osd_map injecting map crc failure 2024-09-13T13:55:35.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-13T13:55:35.892 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:35.890+0000 7f563b88e640 -1 osd.4 220 handle_osd_map injecting map crc failure 2024-09-13T13:55:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T13:55:36.897 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T13:55:37.890 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T13:55:37.907 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T13:55:38.610 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:55:38.609+0000 7f4ffbd7a640 -1 osd.1 222 handle_osd_map injecting map crc failure 2024-09-13T13:55:38.899 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T13:55:38.900 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:55:38.899+0000 7f26bcfe2640 -1 osd.3 224 handle_osd_map injecting map crc failure 2024-09-13T13:55:38.900 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:55:38.899+0000 7fc16c184640 -1 osd.11 224 handle_osd_map injecting map crc failure 2024-09-13T13:55:38.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T13:55:39.909 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T13:55:39.927 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-13T13:55:40.915 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-13T13:55:40.915 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:55:40.915+0000 7f3c37938640 -1 osd.5 226 handle_osd_map injecting map crc failure 2024-09-13T13:55:40.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T13:55:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T13:55:41.937 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-13T13:55:42.543 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:55:42.542+0000 7f26bafde640 -1 osd.3 227 handle_osd_map injecting map crc failure 2024-09-13T13:55:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-13T13:55:42.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T13:55:43.934 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T13:55:43.951 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T13:55:44.940 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T13:55:44.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T13:55:45.013 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:45.013+0000 7f563b88e640 -1 osd.4 229 handle_osd_map injecting map crc failure 2024-09-13T13:55:45.015 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:45.014+0000 7f563b88e640 -1 osd.4 229 handle_osd_map injecting map crc failure 2024-09-13T13:55:45.097 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:55:45.096+0000 7fc16a180640 -1 osd.11 230 handle_osd_map injecting map crc failure 2024-09-13T13:55:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T13:55:45.965 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-13T13:55:46.126 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:55:46.125+0000 7f9b0c553640 -1 osd.12 231 handle_osd_map injecting map crc failure 2024-09-13T13:55:46.580 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:55:46.579+0000 7f8f58c61640 -1 osd.6 231 handle_osd_map injecting map crc failure 2024-09-13T13:55:46.581 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:55:46.580+0000 7f8f58c61640 -1 osd.6 231 handle_osd_map injecting map crc failure 2024-09-13T13:55:46.598 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:55:46.598+0000 7ffaeece8640 -1 osd.0 231 handle_osd_map injecting map crc failure 2024-09-13T13:55:46.957 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T13:55:46.958 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:55:46.956+0000 7f1a526ee640 -1 osd.13 232 handle_osd_map injecting map crc failure 2024-09-13T13:55:46.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T13:55:47.967 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T13:55:47.967 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:55:47.966+0000 7f1a526ee640 -1 osd.13 233 handle_osd_map injecting map crc failure 2024-09-13T13:55:47.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-13T13:55:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-13T13:55:48.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T13:55:49.981 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T13:55:49.983 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:55:49.983+0000 7f7c73c21640 -1 osd.14 235 handle_osd_map injecting map crc failure 2024-09-13T13:55:49.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-13T13:55:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-13T13:55:51.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T13:55:51.116 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:51.115+0000 7fe0f9e03640 -1 osd.2 235 handle_osd_map injecting map crc failure 2024-09-13T13:55:51.996 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T13:55:52.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-13T13:55:52.655 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:55:52.654+0000 7f002dbdb640 -1 osd.15 237 handle_osd_map injecting map crc failure 2024-09-13T13:55:53.003 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T13:55:53.003 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:55:53.002+0000 7f8f5ac65640 -1 osd.6 238 handle_osd_map injecting map crc failure 2024-09-13T13:55:53.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T13:55:54.017 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T13:55:54.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-13T13:55:54.317 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:54.316+0000 7fe0f9e03640 -1 osd.2 239 handle_osd_map injecting map crc failure 2024-09-13T13:55:54.318 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:55:54.316+0000 7f586a715640 -1 osd.9 239 handle_osd_map injecting map crc failure 2024-09-13T13:55:54.318 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:55:54.317+0000 7f26bafde640 -1 osd.3 239 handle_osd_map injecting map crc failure 2024-09-13T13:55:55.021 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T13:55:55.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T13:55:55.416 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:55:55.414+0000 7fe0f9e03640 -1 osd.2 240 handle_osd_map injecting map crc failure 2024-09-13T13:55:55.416 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:55:55.414+0000 7f3c35934640 -1 osd.5 240 handle_osd_map injecting map crc failure 2024-09-13T13:55:56.025 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T13:55:56.025 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:55:56.023+0000 7faeac214640 -1 osd.8 241 handle_osd_map injecting map crc failure 2024-09-13T13:55:56.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-13T13:55:57.035 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-13T13:55:57.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T13:55:58.036 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T13:55:58.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-13T13:55:58.495 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-13T13:55:58.494+0000 7f3c35934640 -1 osd.5 242 handle_osd_map injecting map crc failure 2024-09-13T13:55:58.557 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:55:58.557+0000 7fc16c184640 -1 osd.11 242 handle_osd_map injecting map crc failure 2024-09-13T13:55:58.674 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:55:58.673+0000 7f563b88e640 -1 osd.4 242 handle_osd_map injecting map crc failure 2024-09-13T13:55:59.052 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-13T13:55:59.052 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:55:59.051+0000 7fe859f53640 -1 osd.7 243 handle_osd_map injecting map crc failure 2024-09-13T13:55:59.059 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:55:59.058+0000 7f586c719640 -1 osd.9 243 handle_osd_map injecting map crc failure 2024-09-13T13:55:59.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T13:55:59.382 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:55:59.381+0000 7f8f58c61640 -1 osd.6 243 handle_osd_map injecting map crc failure 2024-09-13T13:56:00.053 INFO:teuthology.orchestra.run.smithi027.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T13:56:00.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-13T13:56:01.062 INFO:teuthology.orchestra.run.smithi027.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T13:56:01.063 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:56:01.062+0000 7f1a526ee640 -1 osd.13 246 handle_osd_map injecting map crc failure 2024-09-13T13:56:01.081 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T13:56:01.081 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T13:56:01.081 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T13:56:01.081 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T13:56:01.081 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T13:56:01.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:56:01.397 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:01.397 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:56:01.060932+0000","last_up_change":"2024-09-13T13:55:17.790049+0000","last_in_change":"2024-09-13T13:55:28.829859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"reef","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T13:50:03.138857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":123,"up_from":211,"up_thru":34,"down_at":124,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":883900485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":883900485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":883900485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":883900485}]},"public_addr":"172.21.15.27:6812/883900485","cluster_addr":"172.21.15.27:6813/883900485","heartbeat_back_addr":"172.21.15.27:6815/883900485","heartbeat_front_addr":"172.21.15.27:6814/883900485","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:55.800399+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:39.778825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:46.317027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:51.629049+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:17.068536+0000","dead_epoch":124},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:53.258839+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:49.792501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:57.209400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:59.453198+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:20.415069+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:48.748942+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:47.627781+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:55.839524+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:52.752930+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:01.069305+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:31.369703+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[{"pgid":"2.2","primary_osd":9}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:56:01.410 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T13:56:01.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:56:01.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:01.715 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":247,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:56:01.060932+0000","last_up_change":"2024-09-13T13:55:17.790049+0000","last_in_change":"2024-09-13T13:55:28.829859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"reef","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T13:50:03.138857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":123,"up_from":211,"up_thru":34,"down_at":124,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":883900485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":883900485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":883900485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":883900485}]},"public_addr":"172.21.15.27:6812/883900485","cluster_addr":"172.21.15.27:6813/883900485","heartbeat_back_addr":"172.21.15.27:6815/883900485","heartbeat_front_addr":"172.21.15.27:6814/883900485","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:55.800399+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:39.778825+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:46.317027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:51.629049+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:17.068536+0000","dead_epoch":124},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:53.258839+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:49.792501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:57.209400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:59.453198+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:20.415069+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:48.748942+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:47.627781+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:55.839524+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:52.752930+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:01.069305+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:31.369703+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[{"pgid":"2.2","primary_osd":9}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:56:01.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T13:56:01.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T13:56:01.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T13:56:01.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T13:56:01.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T13:56:01.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T13:56:01.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T13:56:01.731 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T13:56:01.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T13:56:01.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T13:56:01.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T13:56:01.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T13:56:01.733 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T13:56:01.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T13:56:01.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T13:56:01.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T13:56:02.150 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:02.150 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T13:56:02.164 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:02.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T13:56:02.176 INFO:teuthology.orchestra.run.smithi027.stdout:906238099467 2024-09-13T13:56:02.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T13:56:02.188 INFO:teuthology.orchestra.run.smithi027.stdout:90194313297 2024-09-13T13:56:02.189 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T13:56:02.200 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:02.200 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T13:56:02.212 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:02.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T13:56:02.222 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:02.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T13:56:02.230 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:02.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T13:56:02.235 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:02.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T13:56:02.258 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:02.259 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T13:56:02.259 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:02.260 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:02.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T13:56:02.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T13:56:02.265 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:02.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T13:56:02.278 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:02.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T13:56:02.337 INFO:teuthology.orchestra.run.smithi027.stdout:98784247886 2024-09-13T13:56:02.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T13:56:02.379 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:02.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T13:56:03.276 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:03.296 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:03.308 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:03.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247888 got 98784247888 for osd.0 2024-09-13T13:56:03.315 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.324 INFO:teuthology.orchestra.run.smithi027.stdout:906238099467 2024-09-13T13:56:03.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280591 got 94489280591 for osd.2 2024-09-13T13:56:03.337 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.355 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280591 got 94489280591 for osd.12 2024-09-13T13:56:03.356 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 906238099467 got 906238099467 for osd.4 2024-09-13T13:56:03.366 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:94489280591 2024-09-13T13:56:03.447 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:03.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280591 got 94489280591 for osd.6 2024-09-13T13:56:03.456 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.460 INFO:teuthology.orchestra.run.smithi027.stdout:90194313297 2024-09-13T13:56:03.476 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:03.477 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:03.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.10 2024-09-13T13:56:03.478 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.484 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:03.499 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:03.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247888 got 98784247888 for osd.11 2024-09-13T13:56:03.501 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.502 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313297 got 90194313297 for osd.13 2024-09-13T13:56:03.502 INFO:teuthology.orchestra.run.smithi027.stdout:98784247887 2024-09-13T13:56:03.502 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.9 2024-09-13T13:56:03.509 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.520 INFO:teuthology.orchestra.run.smithi027.stdout:98784247888 2024-09-13T13:56:03.522 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:03.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247887 got 98784247887 for osd.8 2024-09-13T13:56:03.524 INFO:teuthology.orchestra.run.smithi027.stdout:98784247886 2024-09-13T13:56:03.525 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247887 got 98784247887 for osd.15 2024-09-13T13:56:03.526 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247887 got 98784247887 for osd.1 2024-09-13T13:56:03.527 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247888 got 98784247888 for osd.7 2024-09-13T13:56:03.536 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.5 2024-09-13T13:56:03.544 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247886 got 98784247886 for osd.14 2024-09-13T13:56:03.544 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.572 INFO:teuthology.orchestra.run.smithi027.stdout:94489280590 2024-09-13T13:56:03.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280590 got 94489280590 for osd.3 2024-09-13T13:56:03.585 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:03.585 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T13:56:03.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:03.931 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:03.931 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:03.943 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-13T13:56:02.445925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":781144,"kb_used_data":285144,"kb_used_omap":36,"kb_used_meta":495707,"kb_avail":1509168296,"statfs":{"total":1546188226560,"available":1545388335104,"internally_reserved":0,"allocated":291987456,"data_stored":287303143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":507604698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.408862"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572916+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:01.572916+0000","last_peered":"2024-09-13T13:56:01.572916+0000","last_clean":"2024-09-13T13:56:01.572916+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:01.572916+0000","last_undegraded":"2024-09-13T13:56:01.572916+0000","last_fullsized":"2024-09-13T13:56:01.572916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":483,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067280+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:01.067280+0000","last_peered":"2024-09-13T13:56:01.067280+0000","last_clean":"2024-09-13T13:56:01.067280+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:01.067280+0000","last_undegraded":"2024-09-13T13:56:01.067280+0000","last_fullsized":"2024-09-13T13:56:01.067280+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":813,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067251+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:01.067251+0000","last_peered":"2024-09-13T13:56:01.067251+0000","last_clean":"2024-09-13T13:56:01.067251+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:01.067251+0000","last_undegraded":"2024-09-13T13:56:01.067251+0000","last_fullsized":"2024-09-13T13:56:01.067251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572573+0000","last_change":"2024-09-13T13:55:24.307579+0000","last_active":"2024-09-13T13:56:01.572573+0000","last_peered":"2024-09-13T13:56:01.572573+0000","last_clean":"2024-09-13T13:56:01.572573+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:01.572573+0000","last_undegraded":"2024-09-13T13:56:01.572573+0000","last_fullsized":"2024-09-13T13:56:01.572573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:24.307500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:24.307500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:43.868365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":932,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.070030+0000","last_change":"2024-09-13T13:55:30.224247+0000","last_active":"2024-09-13T13:56:01.070030+0000","last_peered":"2024-09-13T13:56:01.070030+0000","last_clean":"2024-09-13T13:56:01.070030+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:01.070030+0000","last_undegraded":"2024-09-13T13:56:01.070030+0000","last_fullsized":"2024-09-13T13:56:01.070030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:54:35.167753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:54:35.167753+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:03.783515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067274+0000","last_change":"2024-09-13T13:54:57.252715+0000","last_active":"2024-09-13T13:56:01.067274+0000","last_peered":"2024-09-13T13:56:01.067274+0000","last_clean":"2024-09-13T13:56:01.067274+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:01.067274+0000","last_undegraded":"2024-09-13T13:56:01.067274+0000","last_fullsized":"2024-09-13T13:56:01.067274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:57.252633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:57.252633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:07.155321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":543,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.424059+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:01.424059+0000","last_peered":"2024-09-13T13:56:01.424059+0000","last_clean":"2024-09-13T13:56:01.424059+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:01.424059+0000","last_undegraded":"2024-09-13T13:56:01.424059+0000","last_fullsized":"2024-09-13T13:56:01.424059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":501,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924631+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:01.924631+0000","last_peered":"2024-09-13T13:56:01.924631+0000","last_clean":"2024-09-13T13:56:01.924631+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:01.924631+0000","last_undegraded":"2024-09-13T13:56:01.924631+0000","last_fullsized":"2024-09-13T13:56:01.924631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":658,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.764576+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:01.764576+0000","last_peered":"2024-09-13T13:56:01.764576+0000","last_clean":"2024-09-13T13:56:01.764576+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:01.764576+0000","last_undegraded":"2024-09-13T13:56:01.764576+0000","last_fullsized":"2024-09-13T13:56:01.764576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.351522+0000","last_change":"2024-09-13T13:55:27.308757+0000","last_active":"2024-09-13T13:56:01.351522+0000","last_peered":"2024-09-13T13:56:01.351522+0000","last_clean":"2024-09-13T13:56:01.351522+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:01.351522+0000","last_undegraded":"2024-09-13T13:56:01.351522+0000","last_fullsized":"2024-09-13T13:56:01.351522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:27.308673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:27.308673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:48.751712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.571386+0000","last_change":"2024-09-13T13:55:02.409900+0000","last_active":"2024-09-13T13:56:01.571386+0000","last_peered":"2024-09-13T13:56:01.571386+0000","last_clean":"2024-09-13T13:56:01.571386+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:01.571386+0000","last_undegraded":"2024-09-13T13:56:01.571386+0000","last_fullsized":"2024-09-13T13:56:01.571386+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:02.409823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:02.409823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:02.679367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924627+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:01.924627+0000","last_peered":"2024-09-13T13:56:01.924627+0000","last_clean":"2024-09-13T13:56:01.924627+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:01.924627+0000","last_undegraded":"2024-09-13T13:56:01.924627+0000","last_fullsized":"2024-09-13T13:56:01.924627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572607+0000","last_change":"2024-09-13T13:55:20.227240+0000","last_active":"2024-09-13T13:56:01.572607+0000","last_peered":"2024-09-13T13:56:01.572607+0000","last_clean":"2024-09-13T13:56:01.572607+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:01.572607+0000","last_undegraded":"2024-09-13T13:56:01.572607+0000","last_fullsized":"2024-09-13T13:56:01.572607+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:20.227146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:20.227146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:27.633347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":1022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43778359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":431139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":430843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105188,"kb_used_data":74004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266652,"statfs":{"total":96636764160,"available":96529051648,"internally_reserved":0,"allocated":75780096,"data_stored":75389370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":425509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":1022185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":431416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60296,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311544,"statfs":{"total":96636764160,"available":96575021056,"internally_reserved":0,"allocated":29810688,"data_stored":29482272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":424619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66704,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":36372480,"data_stored":36046973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":425027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":431683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63740,"kb_used_data":35500,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308100,"statfs":{"total":96636764160,"available":96571494400,"internally_reserved":0,"allocated":36352000,"data_stored":36039135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62772,"kb_used_data":31588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309068,"statfs":{"total":96636764160,"available":96572485632,"internally_reserved":0,"allocated":32346112,"data_stored":32033312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247887,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60308,"kb_used_data":29124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311532,"statfs":{"total":96636764160,"available":96575008768,"internally_reserved":0,"allocated":29822976,"data_stored":29489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:03.944 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:04.247 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:04.260 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-13T13:56:02.445925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":781144,"kb_used_data":285144,"kb_used_omap":36,"kb_used_meta":495707,"kb_avail":1509168296,"statfs":{"total":1546188226560,"available":1545388335104,"internally_reserved":0,"allocated":291987456,"data_stored":287303143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":507604698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.408862"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572916+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:01.572916+0000","last_peered":"2024-09-13T13:56:01.572916+0000","last_clean":"2024-09-13T13:56:01.572916+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:01.572916+0000","last_undegraded":"2024-09-13T13:56:01.572916+0000","last_fullsized":"2024-09-13T13:56:01.572916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":483,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067280+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:01.067280+0000","last_peered":"2024-09-13T13:56:01.067280+0000","last_clean":"2024-09-13T13:56:01.067280+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:01.067280+0000","last_undegraded":"2024-09-13T13:56:01.067280+0000","last_fullsized":"2024-09-13T13:56:01.067280+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":813,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067251+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:01.067251+0000","last_peered":"2024-09-13T13:56:01.067251+0000","last_clean":"2024-09-13T13:56:01.067251+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:01.067251+0000","last_undegraded":"2024-09-13T13:56:01.067251+0000","last_fullsized":"2024-09-13T13:56:01.067251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572573+0000","last_change":"2024-09-13T13:55:24.307579+0000","last_active":"2024-09-13T13:56:01.572573+0000","last_peered":"2024-09-13T13:56:01.572573+0000","last_clean":"2024-09-13T13:56:01.572573+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:01.572573+0000","last_undegraded":"2024-09-13T13:56:01.572573+0000","last_fullsized":"2024-09-13T13:56:01.572573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:24.307500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:24.307500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:43.868365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":932,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.070030+0000","last_change":"2024-09-13T13:55:30.224247+0000","last_active":"2024-09-13T13:56:01.070030+0000","last_peered":"2024-09-13T13:56:01.070030+0000","last_clean":"2024-09-13T13:56:01.070030+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:01.070030+0000","last_undegraded":"2024-09-13T13:56:01.070030+0000","last_fullsized":"2024-09-13T13:56:01.070030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:54:35.167753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:54:35.167753+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:03.783515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067274+0000","last_change":"2024-09-13T13:54:57.252715+0000","last_active":"2024-09-13T13:56:01.067274+0000","last_peered":"2024-09-13T13:56:01.067274+0000","last_clean":"2024-09-13T13:56:01.067274+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:01.067274+0000","last_undegraded":"2024-09-13T13:56:01.067274+0000","last_fullsized":"2024-09-13T13:56:01.067274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:57.252633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:57.252633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:07.155321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":543,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.424059+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:01.424059+0000","last_peered":"2024-09-13T13:56:01.424059+0000","last_clean":"2024-09-13T13:56:01.424059+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:01.424059+0000","last_undegraded":"2024-09-13T13:56:01.424059+0000","last_fullsized":"2024-09-13T13:56:01.424059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":501,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924631+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:01.924631+0000","last_peered":"2024-09-13T13:56:01.924631+0000","last_clean":"2024-09-13T13:56:01.924631+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:01.924631+0000","last_undegraded":"2024-09-13T13:56:01.924631+0000","last_fullsized":"2024-09-13T13:56:01.924631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":658,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.764576+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:01.764576+0000","last_peered":"2024-09-13T13:56:01.764576+0000","last_clean":"2024-09-13T13:56:01.764576+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:01.764576+0000","last_undegraded":"2024-09-13T13:56:01.764576+0000","last_fullsized":"2024-09-13T13:56:01.764576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.351522+0000","last_change":"2024-09-13T13:55:27.308757+0000","last_active":"2024-09-13T13:56:01.351522+0000","last_peered":"2024-09-13T13:56:01.351522+0000","last_clean":"2024-09-13T13:56:01.351522+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:01.351522+0000","last_undegraded":"2024-09-13T13:56:01.351522+0000","last_fullsized":"2024-09-13T13:56:01.351522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:27.308673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:27.308673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:48.751712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.571386+0000","last_change":"2024-09-13T13:55:02.409900+0000","last_active":"2024-09-13T13:56:01.571386+0000","last_peered":"2024-09-13T13:56:01.571386+0000","last_clean":"2024-09-13T13:56:01.571386+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:01.571386+0000","last_undegraded":"2024-09-13T13:56:01.571386+0000","last_fullsized":"2024-09-13T13:56:01.571386+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:02.409823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:02.409823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:02.679367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924627+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:01.924627+0000","last_peered":"2024-09-13T13:56:01.924627+0000","last_clean":"2024-09-13T13:56:01.924627+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:01.924627+0000","last_undegraded":"2024-09-13T13:56:01.924627+0000","last_fullsized":"2024-09-13T13:56:01.924627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572607+0000","last_change":"2024-09-13T13:55:20.227240+0000","last_active":"2024-09-13T13:56:01.572607+0000","last_peered":"2024-09-13T13:56:01.572607+0000","last_clean":"2024-09-13T13:56:01.572607+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:01.572607+0000","last_undegraded":"2024-09-13T13:56:01.572607+0000","last_fullsized":"2024-09-13T13:56:01.572607+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:20.227146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:20.227146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:27.633347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":1022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43778359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":431139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":430843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105188,"kb_used_data":74004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266652,"statfs":{"total":96636764160,"available":96529051648,"internally_reserved":0,"allocated":75780096,"data_stored":75389370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":425509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":1022185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":431416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60296,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311544,"statfs":{"total":96636764160,"available":96575021056,"internally_reserved":0,"allocated":29810688,"data_stored":29482272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":424619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66704,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":36372480,"data_stored":36046973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":425027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":431683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63740,"kb_used_data":35500,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308100,"statfs":{"total":96636764160,"available":96571494400,"internally_reserved":0,"allocated":36352000,"data_stored":36039135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62772,"kb_used_data":31588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309068,"statfs":{"total":96636764160,"available":96572485632,"internally_reserved":0,"allocated":32346112,"data_stored":32033312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247887,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60308,"kb_used_data":29124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311532,"statfs":{"total":96636764160,"available":96575008768,"internally_reserved":0,"allocated":29822976,"data_stored":29489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:04.261 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T13:56:04.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T13:56:04.583 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T13:56:04.592 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:56:04.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:04.894 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:04.895 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:04.907 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-13T13:56:04.446404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":781168,"kb_used_data":285168,"kb_used_omap":36,"kb_used_meta":495707,"kb_avail":1509168272,"statfs":{"total":1546188226560,"available":1545388310528,"internally_reserved":0,"allocated":292012032,"data_stored":287317791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":507604698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001494"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572916+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:01.572916+0000","last_peered":"2024-09-13T13:56:01.572916+0000","last_clean":"2024-09-13T13:56:01.572916+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:01.572916+0000","last_undegraded":"2024-09-13T13:56:01.572916+0000","last_fullsized":"2024-09-13T13:56:01.572916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":483,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067280+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:01.067280+0000","last_peered":"2024-09-13T13:56:01.067280+0000","last_clean":"2024-09-13T13:56:01.067280+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:01.067280+0000","last_undegraded":"2024-09-13T13:56:01.067280+0000","last_fullsized":"2024-09-13T13:56:01.067280+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":813,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067251+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:01.067251+0000","last_peered":"2024-09-13T13:56:01.067251+0000","last_clean":"2024-09-13T13:56:01.067251+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:01.067251+0000","last_undegraded":"2024-09-13T13:56:01.067251+0000","last_fullsized":"2024-09-13T13:56:01.067251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572573+0000","last_change":"2024-09-13T13:55:24.307579+0000","last_active":"2024-09-13T13:56:01.572573+0000","last_peered":"2024-09-13T13:56:01.572573+0000","last_clean":"2024-09-13T13:56:01.572573+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:01.572573+0000","last_undegraded":"2024-09-13T13:56:01.572573+0000","last_fullsized":"2024-09-13T13:56:01.572573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:24.307500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:24.307500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:43.868365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":932,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.070030+0000","last_change":"2024-09-13T13:55:30.224247+0000","last_active":"2024-09-13T13:56:01.070030+0000","last_peered":"2024-09-13T13:56:01.070030+0000","last_clean":"2024-09-13T13:56:01.070030+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:01.070030+0000","last_undegraded":"2024-09-13T13:56:01.070030+0000","last_fullsized":"2024-09-13T13:56:01.070030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:54:35.167753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:54:35.167753+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:03.783515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067274+0000","last_change":"2024-09-13T13:54:57.252715+0000","last_active":"2024-09-13T13:56:01.067274+0000","last_peered":"2024-09-13T13:56:01.067274+0000","last_clean":"2024-09-13T13:56:01.067274+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:01.067274+0000","last_undegraded":"2024-09-13T13:56:01.067274+0000","last_fullsized":"2024-09-13T13:56:01.067274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:57.252633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:57.252633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:07.155321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":543,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.424059+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:01.424059+0000","last_peered":"2024-09-13T13:56:01.424059+0000","last_clean":"2024-09-13T13:56:01.424059+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:01.424059+0000","last_undegraded":"2024-09-13T13:56:01.424059+0000","last_fullsized":"2024-09-13T13:56:01.424059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":501,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924631+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:01.924631+0000","last_peered":"2024-09-13T13:56:01.924631+0000","last_clean":"2024-09-13T13:56:01.924631+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:01.924631+0000","last_undegraded":"2024-09-13T13:56:01.924631+0000","last_fullsized":"2024-09-13T13:56:01.924631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":658,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.764576+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:01.764576+0000","last_peered":"2024-09-13T13:56:01.764576+0000","last_clean":"2024-09-13T13:56:01.764576+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:01.764576+0000","last_undegraded":"2024-09-13T13:56:01.764576+0000","last_fullsized":"2024-09-13T13:56:01.764576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.351522+0000","last_change":"2024-09-13T13:55:27.308757+0000","last_active":"2024-09-13T13:56:01.351522+0000","last_peered":"2024-09-13T13:56:01.351522+0000","last_clean":"2024-09-13T13:56:01.351522+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:01.351522+0000","last_undegraded":"2024-09-13T13:56:01.351522+0000","last_fullsized":"2024-09-13T13:56:01.351522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:27.308673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:27.308673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:48.751712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.571386+0000","last_change":"2024-09-13T13:55:02.409900+0000","last_active":"2024-09-13T13:56:01.571386+0000","last_peered":"2024-09-13T13:56:01.571386+0000","last_clean":"2024-09-13T13:56:01.571386+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:01.571386+0000","last_undegraded":"2024-09-13T13:56:01.571386+0000","last_fullsized":"2024-09-13T13:56:01.571386+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:02.409823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:02.409823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:02.679367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924627+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:01.924627+0000","last_peered":"2024-09-13T13:56:01.924627+0000","last_clean":"2024-09-13T13:56:01.924627+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:01.924627+0000","last_undegraded":"2024-09-13T13:56:01.924627+0000","last_fullsized":"2024-09-13T13:56:01.924627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572607+0000","last_change":"2024-09-13T13:55:20.227240+0000","last_active":"2024-09-13T13:56:01.572607+0000","last_peered":"2024-09-13T13:56:01.572607+0000","last_clean":"2024-09-13T13:56:01.572607+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:01.572607+0000","last_undegraded":"2024-09-13T13:56:01.572607+0000","last_fullsized":"2024-09-13T13:56:01.572607+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:20.227146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:20.227146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:27.633347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":1022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43778359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":431139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":430843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105188,"kb_used_data":74004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266652,"statfs":{"total":96636764160,"available":96529051648,"internally_reserved":0,"allocated":75780096,"data_stored":75389370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":425509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":1022185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":431416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60296,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311544,"statfs":{"total":96636764160,"available":96575021056,"internally_reserved":0,"allocated":29810688,"data_stored":29482272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":424619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66704,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":36372480,"data_stored":36046973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":432351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":431683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63740,"kb_used_data":35500,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308100,"statfs":{"total":96636764160,"available":96571494400,"internally_reserved":0,"allocated":36352000,"data_stored":36039135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62784,"kb_used_data":31600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309056,"statfs":{"total":96636764160,"available":96572473344,"internally_reserved":0,"allocated":32358400,"data_stored":32040636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247888,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60308,"kb_used_data":29124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311532,"statfs":{"total":96636764160,"available":96575008768,"internally_reserved":0,"allocated":29822976,"data_stored":29489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:04.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:05.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:05.212 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:05.223 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-13T13:56:04.446404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":781168,"kb_used_data":285168,"kb_used_omap":36,"kb_used_meta":495707,"kb_avail":1509168272,"statfs":{"total":1546188226560,"available":1545388310528,"internally_reserved":0,"allocated":292012032,"data_stored":287317791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":507604698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001494"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572916+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:01.572916+0000","last_peered":"2024-09-13T13:56:01.572916+0000","last_clean":"2024-09-13T13:56:01.572916+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:01.572916+0000","last_undegraded":"2024-09-13T13:56:01.572916+0000","last_fullsized":"2024-09-13T13:56:01.572916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":483,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067280+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:01.067280+0000","last_peered":"2024-09-13T13:56:01.067280+0000","last_clean":"2024-09-13T13:56:01.067280+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:01.067280+0000","last_undegraded":"2024-09-13T13:56:01.067280+0000","last_fullsized":"2024-09-13T13:56:01.067280+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":813,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067251+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:01.067251+0000","last_peered":"2024-09-13T13:56:01.067251+0000","last_clean":"2024-09-13T13:56:01.067251+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:01.067251+0000","last_undegraded":"2024-09-13T13:56:01.067251+0000","last_fullsized":"2024-09-13T13:56:01.067251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572573+0000","last_change":"2024-09-13T13:55:24.307579+0000","last_active":"2024-09-13T13:56:01.572573+0000","last_peered":"2024-09-13T13:56:01.572573+0000","last_clean":"2024-09-13T13:56:01.572573+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:01.572573+0000","last_undegraded":"2024-09-13T13:56:01.572573+0000","last_fullsized":"2024-09-13T13:56:01.572573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:24.307500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:24.307500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:43.868365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":932,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.070030+0000","last_change":"2024-09-13T13:55:30.224247+0000","last_active":"2024-09-13T13:56:01.070030+0000","last_peered":"2024-09-13T13:56:01.070030+0000","last_clean":"2024-09-13T13:56:01.070030+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:01.070030+0000","last_undegraded":"2024-09-13T13:56:01.070030+0000","last_fullsized":"2024-09-13T13:56:01.070030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:54:35.167753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:54:35.167753+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:03.783515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067274+0000","last_change":"2024-09-13T13:54:57.252715+0000","last_active":"2024-09-13T13:56:01.067274+0000","last_peered":"2024-09-13T13:56:01.067274+0000","last_clean":"2024-09-13T13:56:01.067274+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:01.067274+0000","last_undegraded":"2024-09-13T13:56:01.067274+0000","last_fullsized":"2024-09-13T13:56:01.067274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:57.252633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:57.252633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:07.155321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":543,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.424059+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:01.424059+0000","last_peered":"2024-09-13T13:56:01.424059+0000","last_clean":"2024-09-13T13:56:01.424059+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:01.424059+0000","last_undegraded":"2024-09-13T13:56:01.424059+0000","last_fullsized":"2024-09-13T13:56:01.424059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":501,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924631+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:01.924631+0000","last_peered":"2024-09-13T13:56:01.924631+0000","last_clean":"2024-09-13T13:56:01.924631+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:01.924631+0000","last_undegraded":"2024-09-13T13:56:01.924631+0000","last_fullsized":"2024-09-13T13:56:01.924631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":658,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.764576+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:01.764576+0000","last_peered":"2024-09-13T13:56:01.764576+0000","last_clean":"2024-09-13T13:56:01.764576+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:01.764576+0000","last_undegraded":"2024-09-13T13:56:01.764576+0000","last_fullsized":"2024-09-13T13:56:01.764576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.351522+0000","last_change":"2024-09-13T13:55:27.308757+0000","last_active":"2024-09-13T13:56:01.351522+0000","last_peered":"2024-09-13T13:56:01.351522+0000","last_clean":"2024-09-13T13:56:01.351522+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:01.351522+0000","last_undegraded":"2024-09-13T13:56:01.351522+0000","last_fullsized":"2024-09-13T13:56:01.351522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:27.308673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:27.308673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:48.751712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.571386+0000","last_change":"2024-09-13T13:55:02.409900+0000","last_active":"2024-09-13T13:56:01.571386+0000","last_peered":"2024-09-13T13:56:01.571386+0000","last_clean":"2024-09-13T13:56:01.571386+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:01.571386+0000","last_undegraded":"2024-09-13T13:56:01.571386+0000","last_fullsized":"2024-09-13T13:56:01.571386+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:02.409823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:02.409823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:02.679367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924627+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:01.924627+0000","last_peered":"2024-09-13T13:56:01.924627+0000","last_clean":"2024-09-13T13:56:01.924627+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:01.924627+0000","last_undegraded":"2024-09-13T13:56:01.924627+0000","last_fullsized":"2024-09-13T13:56:01.924627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572607+0000","last_change":"2024-09-13T13:55:20.227240+0000","last_active":"2024-09-13T13:56:01.572607+0000","last_peered":"2024-09-13T13:56:01.572607+0000","last_clean":"2024-09-13T13:56:01.572607+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:01.572607+0000","last_undegraded":"2024-09-13T13:56:01.572607+0000","last_fullsized":"2024-09-13T13:56:01.572607+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:20.227146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:20.227146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:27.633347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":1022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43778359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":431139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":430843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105188,"kb_used_data":74004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266652,"statfs":{"total":96636764160,"available":96529051648,"internally_reserved":0,"allocated":75780096,"data_stored":75389370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":425509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":1022185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":431416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60296,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311544,"statfs":{"total":96636764160,"available":96575021056,"internally_reserved":0,"allocated":29810688,"data_stored":29482272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":424619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66704,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":36372480,"data_stored":36046973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":432351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":431683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63740,"kb_used_data":35500,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308100,"statfs":{"total":96636764160,"available":96571494400,"internally_reserved":0,"allocated":36352000,"data_stored":36039135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62784,"kb_used_data":31600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309056,"statfs":{"total":96636764160,"available":96572473344,"internally_reserved":0,"allocated":32358400,"data_stored":32040636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247888,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60308,"kb_used_data":29124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311532,"statfs":{"total":96636764160,"available":96575008768,"internally_reserved":0,"allocated":29822976,"data_stored":29489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:05.224 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:56:05.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:05.524 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:05.524 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:05.536 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-13T13:56:04.446404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":781168,"kb_used_data":285168,"kb_used_omap":36,"kb_used_meta":495707,"kb_avail":1509168272,"statfs":{"total":1546188226560,"available":1545388310528,"internally_reserved":0,"allocated":292012032,"data_stored":287317791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":507604698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001494"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572916+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:01.572916+0000","last_peered":"2024-09-13T13:56:01.572916+0000","last_clean":"2024-09-13T13:56:01.572916+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:01.572916+0000","last_undegraded":"2024-09-13T13:56:01.572916+0000","last_fullsized":"2024-09-13T13:56:01.572916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":483,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067280+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:01.067280+0000","last_peered":"2024-09-13T13:56:01.067280+0000","last_clean":"2024-09-13T13:56:01.067280+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:01.067280+0000","last_undegraded":"2024-09-13T13:56:01.067280+0000","last_fullsized":"2024-09-13T13:56:01.067280+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":813,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067251+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:01.067251+0000","last_peered":"2024-09-13T13:56:01.067251+0000","last_clean":"2024-09-13T13:56:01.067251+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:01.067251+0000","last_undegraded":"2024-09-13T13:56:01.067251+0000","last_fullsized":"2024-09-13T13:56:01.067251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572573+0000","last_change":"2024-09-13T13:55:24.307579+0000","last_active":"2024-09-13T13:56:01.572573+0000","last_peered":"2024-09-13T13:56:01.572573+0000","last_clean":"2024-09-13T13:56:01.572573+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:01.572573+0000","last_undegraded":"2024-09-13T13:56:01.572573+0000","last_fullsized":"2024-09-13T13:56:01.572573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:24.307500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:24.307500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:43.868365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":932,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.070030+0000","last_change":"2024-09-13T13:55:30.224247+0000","last_active":"2024-09-13T13:56:01.070030+0000","last_peered":"2024-09-13T13:56:01.070030+0000","last_clean":"2024-09-13T13:56:01.070030+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:01.070030+0000","last_undegraded":"2024-09-13T13:56:01.070030+0000","last_fullsized":"2024-09-13T13:56:01.070030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:54:35.167753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:54:35.167753+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:03.783515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.067274+0000","last_change":"2024-09-13T13:54:57.252715+0000","last_active":"2024-09-13T13:56:01.067274+0000","last_peered":"2024-09-13T13:56:01.067274+0000","last_clean":"2024-09-13T13:56:01.067274+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:01.067274+0000","last_undegraded":"2024-09-13T13:56:01.067274+0000","last_fullsized":"2024-09-13T13:56:01.067274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:57.252633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:57.252633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:07.155321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":543,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.424059+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:01.424059+0000","last_peered":"2024-09-13T13:56:01.424059+0000","last_clean":"2024-09-13T13:56:01.424059+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:01.424059+0000","last_undegraded":"2024-09-13T13:56:01.424059+0000","last_fullsized":"2024-09-13T13:56:01.424059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":501,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924631+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:01.924631+0000","last_peered":"2024-09-13T13:56:01.924631+0000","last_clean":"2024-09-13T13:56:01.924631+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:01.924631+0000","last_undegraded":"2024-09-13T13:56:01.924631+0000","last_fullsized":"2024-09-13T13:56:01.924631+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":658,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.764576+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:01.764576+0000","last_peered":"2024-09-13T13:56:01.764576+0000","last_clean":"2024-09-13T13:56:01.764576+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:01.764576+0000","last_undegraded":"2024-09-13T13:56:01.764576+0000","last_fullsized":"2024-09-13T13:56:01.764576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.351522+0000","last_change":"2024-09-13T13:55:27.308757+0000","last_active":"2024-09-13T13:56:01.351522+0000","last_peered":"2024-09-13T13:56:01.351522+0000","last_clean":"2024-09-13T13:56:01.351522+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:01.351522+0000","last_undegraded":"2024-09-13T13:56:01.351522+0000","last_fullsized":"2024-09-13T13:56:01.351522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:27.308673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:27.308673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:48.751712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.571386+0000","last_change":"2024-09-13T13:55:02.409900+0000","last_active":"2024-09-13T13:56:01.571386+0000","last_peered":"2024-09-13T13:56:01.571386+0000","last_clean":"2024-09-13T13:56:01.571386+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:01.571386+0000","last_undegraded":"2024-09-13T13:56:01.571386+0000","last_fullsized":"2024-09-13T13:56:01.571386+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:02.409823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:02.409823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:02.679367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":481,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.924627+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:01.924627+0000","last_peered":"2024-09-13T13:56:01.924627+0000","last_clean":"2024-09-13T13:56:01.924627+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:01.924627+0000","last_undegraded":"2024-09-13T13:56:01.924627+0000","last_fullsized":"2024-09-13T13:56:01.924627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":488,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T13:56:01.572607+0000","last_change":"2024-09-13T13:55:20.227240+0000","last_active":"2024-09-13T13:56:01.572607+0000","last_peered":"2024-09-13T13:56:01.572607+0000","last_clean":"2024-09-13T13:56:01.572607+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:01.572607+0000","last_undegraded":"2024-09-13T13:56:01.572607+0000","last_fullsized":"2024-09-13T13:56:01.572607+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:55:20.227146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:20.227146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:27.633347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":1022022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43778359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":431139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31848,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":679936,"data_stored":430843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105188,"kb_used_data":74004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266652,"statfs":{"total":96636764160,"available":96529051648,"internally_reserved":0,"allocated":75780096,"data_stored":75389370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":425509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":1022185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280590,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":431416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60296,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311544,"statfs":{"total":96636764160,"available":96575021056,"internally_reserved":0,"allocated":29810688,"data_stored":29482272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":424619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66704,"kb_used_data":35520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305136,"statfs":{"total":96636764160,"available":96568459264,"internally_reserved":0,"allocated":36372480,"data_stored":36046973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":432351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":431683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63740,"kb_used_data":35500,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308100,"statfs":{"total":96636764160,"available":96571494400,"internally_reserved":0,"allocated":36352000,"data_stored":36039135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62784,"kb_used_data":31600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309056,"statfs":{"total":96636764160,"available":96572473344,"internally_reserved":0,"allocated":32358400,"data_stored":32040636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247888,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60308,"kb_used_data":29124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311532,"statfs":{"total":96636764160,"available":96575008768,"internally_reserved":0,"allocated":29822976,"data_stored":29489279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:05.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:56:05.654 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:56:05.653+0000 7f1a506ea640 -1 osd.13 247 handle_osd_map injecting map crc failure 2024-09-13T13:56:05.843 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:05.843 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":248,"fsid":"cd1b3cf6-d610-4c64-94c3-504da850dd35","created":"2024-09-13T13:49:33.733123+0000","modified":"2024-09-13T13:56:05.568186+0000","last_up_change":"2024-09-13T13:55:17.790049+0000","last_in_change":"2024-09-13T13:55:28.829859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"reef","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:49:46.385182+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:49:48.650814+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T13:50:03.138857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1259d7f4-8ac0-4fdc-8473-9bdafc036502","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":4285669868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6809","nonce":4285669868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6811","nonce":4285669868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":4285669868}]},"public_addr":"172.21.15.27:6808/4285669868","cluster_addr":"172.21.15.27:6809/4285669868","heartbeat_back_addr":"172.21.15.27:6811/4285669868","heartbeat_front_addr":"172.21.15.27:6810/4285669868","state":["exists","up"]},{"osd":1,"uuid":"9035b2e5-ff53-4675-83b8-bf76ab2d3735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":4081030369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6809","nonce":4081030369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6811","nonce":4081030369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":4081030369}]},"public_addr":"172.21.15.100:6808/4081030369","cluster_addr":"172.21.15.100:6809/4081030369","heartbeat_back_addr":"172.21.15.100:6811/4081030369","heartbeat_front_addr":"172.21.15.100:6810/4081030369","state":["exists","up"]},{"osd":2,"uuid":"497bd315-4b6e-4f41-b0de-488a942ac4d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2454012715}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6801","nonce":2454012715}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":2454012715}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2454012715}]},"public_addr":"172.21.15.122:6800/2454012715","cluster_addr":"172.21.15.122:6801/2454012715","heartbeat_back_addr":"172.21.15.122:6803/2454012715","heartbeat_front_addr":"172.21.15.122:6802/2454012715","state":["exists","up"]},{"osd":3,"uuid":"024edf52-c84f-4d52-b825-c203ed5aecb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6801","nonce":3322176894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":3322176894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":3322176894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":3322176894}]},"public_addr":"172.21.15.159:6801/3322176894","cluster_addr":"172.21.15.159:6802/3322176894","heartbeat_back_addr":"172.21.15.159:6804/3322176894","heartbeat_front_addr":"172.21.15.159:6803/3322176894","state":["exists","up"]},{"osd":4,"uuid":"c440743d-dac9-4c75-b6b3-94360a48f75c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":23,"last_clean_end":123,"up_from":211,"up_thru":34,"down_at":124,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":883900485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6813","nonce":883900485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6815","nonce":883900485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":883900485}]},"public_addr":"172.21.15.27:6812/883900485","cluster_addr":"172.21.15.27:6813/883900485","heartbeat_back_addr":"172.21.15.27:6815/883900485","heartbeat_front_addr":"172.21.15.27:6814/883900485","state":["exists","up"]},{"osd":5,"uuid":"d8203cad-54c8-43f1-af73-ec199c8a41c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3923508322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6805","nonce":3923508322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6807","nonce":3923508322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3923508322}]},"public_addr":"172.21.15.100:6804/3923508322","cluster_addr":"172.21.15.100:6805/3923508322","heartbeat_back_addr":"172.21.15.100:6807/3923508322","heartbeat_front_addr":"172.21.15.100:6806/3923508322","state":["exists","up"]},{"osd":6,"uuid":"d5e9b934-6b1e-4f27-838c-6529f69a5b65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1372880522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":1372880522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6807","nonce":1372880522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1372880522}]},"public_addr":"172.21.15.122:6804/1372880522","cluster_addr":"172.21.15.122:6805/1372880522","heartbeat_back_addr":"172.21.15.122:6807/1372880522","heartbeat_front_addr":"172.21.15.122:6806/1372880522","state":["exists","up"]},{"osd":7,"uuid":"46d21c0b-bd67-45c9-8bcc-9e76edd03769","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":1499158947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":1499158947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6816","nonce":1499158947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6815","nonce":1499158947}]},"public_addr":"172.21.15.159:6813/1499158947","cluster_addr":"172.21.15.159:6814/1499158947","heartbeat_back_addr":"172.21.15.159:6816/1499158947","heartbeat_front_addr":"172.21.15.159:6815/1499158947","state":["exists","up"]},{"osd":8,"uuid":"bc60a0ec-af2e-4cd1-abb1-2ac49909a1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":330880331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6805","nonce":330880331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":330880331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":330880331}]},"public_addr":"172.21.15.27:6804/330880331","cluster_addr":"172.21.15.27:6805/330880331","heartbeat_back_addr":"172.21.15.27:6807/330880331","heartbeat_front_addr":"172.21.15.27:6806/330880331","state":["exists","up"]},{"osd":9,"uuid":"ed6ed2c8-c1d2-4e00-a42c-0d8020552b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1204111774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6813","nonce":1204111774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6815","nonce":1204111774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1204111774}]},"public_addr":"172.21.15.100:6812/1204111774","cluster_addr":"172.21.15.100:6813/1204111774","heartbeat_back_addr":"172.21.15.100:6815/1204111774","heartbeat_front_addr":"172.21.15.100:6814/1204111774","state":["exists","up"]},{"osd":10,"uuid":"417658a1-1270-435c-8e2a-b9e4a8b0dba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2574350402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":2574350402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":2574350402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2574350402}]},"public_addr":"172.21.15.122:6812/2574350402","cluster_addr":"172.21.15.122:6813/2574350402","heartbeat_back_addr":"172.21.15.122:6815/2574350402","heartbeat_front_addr":"172.21.15.122:6814/2574350402","state":["exists","up"]},{"osd":11,"uuid":"9191abab-883f-40a1-8176-3c2e74d90c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":1984230453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6810","nonce":1984230453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":1984230453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":1984230453}]},"public_addr":"172.21.15.159:6809/1984230453","cluster_addr":"172.21.15.159:6810/1984230453","heartbeat_back_addr":"172.21.15.159:6812/1984230453","heartbeat_front_addr":"172.21.15.159:6811/1984230453","state":["exists","up"]},{"osd":12,"uuid":"b9a2390c-9646-489d-b46b-268aa3e73194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2410035394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6801","nonce":2410035394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":2410035394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":2410035394}]},"public_addr":"172.21.15.27:6800/2410035394","cluster_addr":"172.21.15.27:6801/2410035394","heartbeat_back_addr":"172.21.15.27:6803/2410035394","heartbeat_front_addr":"172.21.15.27:6802/2410035394","state":["exists","up"]},{"osd":13,"uuid":"b379c5e8-23a1-4fce-8da7-f6cbdbf4fa18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":450451503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6801","nonce":450451503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6803","nonce":450451503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":450451503}]},"public_addr":"172.21.15.100:6800/450451503","cluster_addr":"172.21.15.100:6801/450451503","heartbeat_back_addr":"172.21.15.100:6803/450451503","heartbeat_front_addr":"172.21.15.100:6802/450451503","state":["exists","up"]},{"osd":14,"uuid":"99ad3153-7823-4dd9-9c09-044eba14bcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":1255553857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6809","nonce":1255553857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6811","nonce":1255553857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":1255553857}]},"public_addr":"172.21.15.122:6808/1255553857","cluster_addr":"172.21.15.122:6809/1255553857","heartbeat_back_addr":"172.21.15.122:6811/1255553857","heartbeat_front_addr":"172.21.15.122:6810/1255553857","state":["exists","up"]},{"osd":15,"uuid":"0880b9c1-ac74-4794-b193-f21d0f1a27f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":4073091144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":4073091144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":4073091144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":4073091144}]},"public_addr":"172.21.15.159:6805/4073091144","cluster_addr":"172.21.15.159:6806/4073091144","heartbeat_back_addr":"172.21.15.159:6808/4073091144","heartbeat_front_addr":"172.21.15.159:6807/4073091144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:55.800399+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:56:04.955638+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:46.317027+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:51.629049+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:17.068536+0000","dead_epoch":124},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:53.258839+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:49.792501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:57.209400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:54:59.453198+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:20.415069+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:48.748942+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:47.627781+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:55.839524+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:52.752930+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:01.069305+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:55:31.369703+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[{"pgid":"2.2","primary_osd":9}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:56:06.383 INFO:tasks.ceph.osd.15.smithi159.stderr:2024-09-13T13:56:06.382+0000 7f002dbdb640 -1 osd.15 247 handle_osd_map injecting map crc failure 2024-09-13T13:56:06.857 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T13:56:06.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:07.010 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:07.010 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:07.011 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:07.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T13:56:07.327 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T13:56:07.341 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T13:56:07.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:07.477 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:07.477 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:07.477 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:07.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T13:56:07.794 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T13:56:07.807 INFO:tasks.ceph:Scrubbing osd.8 2024-09-13T13:56:07.808 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:07.945 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:07.945 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:07.945 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:07.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-13T13:56:08.257 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 8 to deep-scrub 2024-09-13T13:56:08.270 INFO:tasks.ceph:Scrubbing osd.12 2024-09-13T13:56:08.271 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:08.408 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:08.408 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:08.408 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:08.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-13T13:56:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 12 to deep-scrub 2024-09-13T13:56:08.739 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T13:56:08.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:08.851 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-09-13T13:56:08.849+0000 7f4ffbd7a640 -1 osd.1 248 handle_osd_map injecting map crc failure 2024-09-13T13:56:08.878 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:08.878 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:08.878 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:08.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T13:56:08.939 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:56:08.939+0000 7f26bafde640 -1 osd.3 248 handle_osd_map injecting map crc failure 2024-09-13T13:56:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T13:56:09.209 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T13:56:09.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:09.347 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:09.348 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:09.348 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:09.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T13:56:09.664 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T13:56:09.676 INFO:tasks.ceph:Scrubbing osd.9 2024-09-13T13:56:09.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:09.816 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:09.816 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:09.816 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:09.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-13T13:56:10.130 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 9 to deep-scrub 2024-09-13T13:56:10.143 INFO:tasks.ceph:Scrubbing osd.13 2024-09-13T13:56:10.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:10.282 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:10.282 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:10.282 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:10.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-13T13:56:10.599 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 13 to deep-scrub 2024-09-13T13:56:10.613 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T13:56:10.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:10.753 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:10.753 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:10.753 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:10.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T13:56:11.070 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T13:56:11.084 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T13:56:11.084 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:11.151 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:56:11.150+0000 7faeaa210640 -1 osd.8 249 handle_osd_map injecting map crc failure 2024-09-13T13:56:11.215 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:56:11.214+0000 7f26bafde640 -1 osd.3 249 handle_osd_map injecting map crc failure 2024-09-13T13:56:11.222 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:11.222 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:11.223 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:11.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T13:56:11.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-13T13:56:11.539+0000 7ff311f9a640 0 --2- 172.21.15.27:0/1268491777 >> v2:172.21.15.27:3300/0 conn(0x7ff30c152880 0x7ff30c0cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T13:56:11.598 INFO:tasks.ceph.osd.6.smithi122.stderr:2024-09-13T13:56:11.597+0000 7f8f5ac65640 -1 osd.6 250 handle_osd_map injecting map crc failure 2024-09-13T13:56:11.752 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T13:56:11.766 INFO:tasks.ceph:Scrubbing osd.10 2024-09-13T13:56:11.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:11.905 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:11.905 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:11.905 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:11.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-13T13:56:12.221 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 10 to deep-scrub 2024-09-13T13:56:12.234 INFO:tasks.ceph:Scrubbing osd.14 2024-09-13T13:56:12.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:12.373 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:12.373 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:12.373 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:12.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-13T13:56:12.692 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 14 to deep-scrub 2024-09-13T13:56:12.705 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T13:56:12.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:12.843 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:12.844 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:12.844 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:12.852 INFO:tasks.ceph.osd.9.smithi100.stderr:2024-09-13T13:56:12.851+0000 7f586a715640 -1 osd.9 251 handle_osd_map injecting map crc failure 2024-09-13T13:56:12.852 INFO:tasks.ceph.osd.12.smithi027.stderr:2024-09-13T13:56:12.851+0000 7f9b0c553640 -1 osd.12 251 handle_osd_map injecting map crc failure 2024-09-13T13:56:12.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T13:56:12.858 INFO:tasks.ceph.osd.13.smithi100.stderr:2024-09-13T13:56:12.858+0000 7f1a506ea640 -1 osd.13 251 handle_osd_map injecting map crc failure 2024-09-13T13:56:13.159 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T13:56:13.172 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T13:56:13.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:13.309 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:13.309 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:13.309 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:13.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T13:56:13.632 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T13:56:13.645 INFO:tasks.ceph:Scrubbing osd.11 2024-09-13T13:56:13.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:13.781 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:13.782 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:13.782 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:13.792 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-13T13:56:14.097 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 11 to deep-scrub 2024-09-13T13:56:14.110 INFO:tasks.ceph:Scrubbing osd.15 2024-09-13T13:56:14.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:56:14.248 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-13T13:56:14.248 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:56:14.248 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-13T13:56:14.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-13T13:56:14.561 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 15 to deep-scrub 2024-09-13T13:56:14.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:14.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:14.881 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:14.893 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-13T13:56:14.448394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":784604,"kb_used_data":284444,"kb_used_omap":36,"kb_used_meta":499867,"kb_avail":1509164836,"statfs":{"total":1546188226560,"available":1545384792064,"internally_reserved":0,"allocated":291270656,"data_stored":286876109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":511864538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.865775"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":589,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T13:56:11.606191+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:11.606191+0000","last_peered":"2024-09-13T13:56:11.606191+0000","last_clean":"2024-09-13T13:56:11.606191+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:11.606191+0000","last_undegraded":"2024-09-13T13:56:11.606191+0000","last_fullsized":"2024-09-13T13:56:11.606191+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":492,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.173556+0000","last_change":"2024-09-13T13:55:22.179942+0000","last_active":"2024-09-13T13:56:12.173556+0000","last_peered":"2024-09-13T13:56:12.173556+0000","last_clean":"2024-09-13T13:56:12.173556+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:12.173556+0000","last_undegraded":"2024-09-13T13:56:12.173556+0000","last_fullsized":"2024-09-13T13:56:12.173556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T13:55:22.179836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:22.179836+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:34.880595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"171'180","reported_seq":821,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.173544+0000","last_change":"2024-09-13T13:55:28.258910+0000","last_active":"2024-09-13T13:56:12.173544+0000","last_peered":"2024-09-13T13:56:12.173544+0000","last_clean":"2024-09-13T13:56:12.173544+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:12.173544+0000","last_undegraded":"2024-09-13T13:56:12.173544+0000","last_fullsized":"2024-09-13T13:56:12.173544+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'180","last_scrub_stamp":"2024-09-13T13:55:28.258795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.258795+0000","objects_scrubbed":43,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:36.755661+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":507,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.854350+0000","last_change":"2024-09-13T13:56:08.940307+0000","last_active":"2024-09-13T13:56:12.854350+0000","last_peered":"2024-09-13T13:56:12.854350+0000","last_clean":"2024-09-13T13:56:12.854350+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:12.854350+0000","last_undegraded":"2024-09-13T13:56:12.854350+0000","last_fullsized":"2024-09-13T13:56:12.854350+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_clean_scrub_stamp":"2024-09-13T13:56:08.940224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:27.740256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"172'175","reported_seq":946,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T13:56:10.671633+0000","last_change":"2024-09-13T13:56:05.659743+0000","last_active":"2024-09-13T13:56:10.671633+0000","last_peered":"2024-09-13T13:56:10.671633+0000","last_clean":"2024-09-13T13:56:10.671633+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:10.671633+0000","last_undegraded":"2024-09-13T13:56:10.671633+0000","last_fullsized":"2024-09-13T13:56:10.671633+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'175","last_scrub_stamp":"2024-09-13T13:56:05.659637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:56:05.659637+0000","objects_scrubbed":41,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:25.735755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":499,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.173544+0000","last_change":"2024-09-13T13:56:07.357180+0000","last_active":"2024-09-13T13:56:12.173544+0000","last_peered":"2024-09-13T13:56:12.173544+0000","last_clean":"2024-09-13T13:56:12.173544+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:12.173544+0000","last_undegraded":"2024-09-13T13:56:12.173544+0000","last_fullsized":"2024-09-13T13:56:12.173544+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:07.357104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:56:07.357104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:25.849480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"172'155","reported_seq":547,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T13:56:08.945644+0000","last_change":"2024-09-13T13:55:29.273987+0000","last_active":"2024-09-13T13:56:08.945644+0000","last_peered":"2024-09-13T13:56:08.945644+0000","last_clean":"2024-09-13T13:56:08.945644+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:08.945644+0000","last_undegraded":"2024-09-13T13:56:08.945644+0000","last_fullsized":"2024-09-13T13:56:08.945644+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'155","last_scrub_stamp":"2024-09-13T13:55:29.273882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.273882+0000","objects_scrubbed":36,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:55.785682+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":505,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T13:56:09.526840+0000","last_change":"2024-09-13T13:55:16.183198+0000","last_active":"2024-09-13T13:56:09.526840+0000","last_peered":"2024-09-13T13:56:09.526840+0000","last_clean":"2024-09-13T13:56:09.526840+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:09.526840+0000","last_undegraded":"2024-09-13T13:56:09.526840+0000","last_fullsized":"2024-09-13T13:56:09.526840+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T13:55:16.183100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:55:16.183100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:39.696540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"170'154","reported_seq":664,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T13:56:10.597336+0000","last_change":"2024-09-13T13:55:28.269943+0000","last_active":"2024-09-13T13:56:10.597336+0000","last_peered":"2024-09-13T13:56:10.597336+0000","last_clean":"2024-09-13T13:56:10.597336+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:10.597336+0000","last_undegraded":"2024-09-13T13:56:10.597336+0000","last_fullsized":"2024-09-13T13:56:10.597336+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'154","last_scrub_stamp":"2024-09-13T13:55:28.269829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:50:04.058166+0000","last_clean_scrub_stamp":"2024-09-13T13:55:28.269829+0000","objects_scrubbed":33,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:51.499073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":505,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.605619+0000","last_change":"2024-09-13T13:56:10.152629+0000","last_active":"2024-09-13T13:56:12.605619+0000","last_peered":"2024-09-13T13:56:12.605619+0000","last_clean":"2024-09-13T13:56:12.605619+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:12.605619+0000","last_undegraded":"2024-09-13T13:56:12.605619+0000","last_fullsized":"2024-09-13T13:56:12.605619+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_clean_scrub_stamp":"2024-09-13T13:56:10.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:32.489358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":505,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.853957+0000","last_change":"2024-09-13T13:56:03.153182+0000","last_active":"2024-09-13T13:56:12.853957+0000","last_peered":"2024-09-13T13:56:12.853957+0000","last_clean":"2024-09-13T13:56:12.853957+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:12.853957+0000","last_undegraded":"2024-09-13T13:56:12.853957+0000","last_fullsized":"2024-09-13T13:56:12.853957+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:03.153097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:56:03.153097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:10.428819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":485,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T13:56:09.526840+0000","last_change":"2024-09-13T13:54:58.355274+0000","last_active":"2024-09-13T13:56:09.526840+0000","last_peered":"2024-09-13T13:56:09.526840+0000","last_clean":"2024-09-13T13:56:09.526840+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:09.526840+0000","last_undegraded":"2024-09-13T13:56:09.526840+0000","last_fullsized":"2024-09-13T13:56:09.526840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:54:58.355182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:49.210716+0000","last_clean_scrub_stamp":"2024-09-13T13:54:58.355182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:11.858258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":506,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T13:56:12.854378+0000","last_change":"2024-09-13T13:56:09.968711+0000","last_active":"2024-09-13T13:56:12.854378+0000","last_peered":"2024-09-13T13:56:12.854378+0000","last_clean":"2024-09-13T13:56:12.854378+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:12.854378+0000","last_undegraded":"2024-09-13T13:56:12.854378+0000","last_fullsized":"2024-09-13T13:56:12.854378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_clean_scrub_stamp":"2024-09-13T13:56:09.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:14.050552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":664,"ondisk_log_size":664,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":978172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74168,"kb_used_data":42984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297672,"statfs":{"total":96636764160,"available":96560816128,"internally_reserved":0,"allocated":44015616,"data_stored":43735147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":386995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":445575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280592,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105088,"kb_used_data":73904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266752,"statfs":{"total":96636764160,"available":96529154048,"internally_reserved":0,"allocated":75677696,"data_stored":75330788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31900,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":733184,"data_stored":447565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280594,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1314816,"data_stored":1036917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280592,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":589824,"data_stored":372834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60332,"kb_used_data":29148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311508,"statfs":{"total":96636764160,"available":96574984192,"internally_reserved":0,"allocated":29847552,"data_stored":29504328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31764,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340076,"statfs":{"total":96636764160,"available":96604237824,"internally_reserved":0,"allocated":593920,"data_stored":373403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66724,"kb_used_data":35540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305116,"statfs":{"total":96636764160,"available":96568438784,"internally_reserved":0,"allocated":36392960,"data_stored":36061411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":388845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31776,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":606208,"data_stored":380761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099469,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63684,"kb_used_data":35444,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308156,"statfs":{"total":96636764160,"available":96571551744,"internally_reserved":0,"allocated":36294656,"data_stored":35990271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62708,"kb_used_data":31524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309132,"statfs":{"total":96636764160,"available":96572551168,"internally_reserved":0,"allocated":32280576,"data_stored":31990302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64404,"kb_used_data":29060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307436,"statfs":{"total":96636764160,"available":96570814464,"internally_reserved":0,"allocated":29757440,"data_stored":29452795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:14.897 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T13:55:29.093390+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=29, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.897 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T13:55:22.179836+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=22, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.897 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T13:55:28.258795+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=28, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.897 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-13T13:56:05.659637+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.897 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-13T13:55:29.273882+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=29, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.898 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T13:55:16.183100+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=16, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.898 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-13T13:55:28.269829+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=28, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.898 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T13:56:03.153097+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=3, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.898 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T13:54:58.355182+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=54, tm_sec=58, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:14.898 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T13:56:17.331 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:56:17.329+0000 7f563b88e640 -1 osd.4 252 handle_osd_map injecting map crc failure 2024-09-13T13:56:21.872 INFO:tasks.ceph.osd.8.smithi027.stderr:2024-09-13T13:56:21.872+0000 7faeaa210640 -1 osd.8 253 handle_osd_map injecting map crc failure 2024-09-13T13:56:21.873 INFO:tasks.ceph.osd.3.smithi159.stderr:2024-09-13T13:56:21.871+0000 7f26bafde640 -1 osd.3 253 handle_osd_map injecting map crc failure 2024-09-13T13:56:25.763 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-09-13T13:56:25.762+0000 7fe0f9e03640 -1 osd.2 254 handle_osd_map injecting map crc failure 2024-09-13T13:56:25.763 INFO:tasks.ceph.osd.10.smithi122.stderr:2024-09-13T13:56:25.762+0000 7f2282f0c640 -1 osd.10 254 handle_osd_map injecting map crc failure 2024-09-13T13:56:25.763 INFO:tasks.ceph.osd.11.smithi159.stderr:2024-09-13T13:56:25.761+0000 7fc16a180640 -1 osd.11 254 handle_osd_map injecting map crc failure 2024-09-13T13:56:26.705 INFO:tasks.ceph.osd.14.smithi122.stderr:2024-09-13T13:56:26.705+0000 7f7c71c1d640 -1 osd.14 254 handle_osd_map injecting map crc failure 2024-09-13T13:56:34.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:35.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:35.215 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:35.228 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-13T13:56:34.453595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":784892,"kb_used_data":284732,"kb_used_omap":36,"kb_used_meta":499867,"kb_avail":1509164548,"statfs":{"total":1546188226560,"available":1545384497152,"internally_reserved":0,"allocated":291565568,"data_stored":287114649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":511864538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002925"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":598,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:24.832827+0000","last_change":"2024-09-13T13:55:29.093477+0000","last_active":"2024-09-13T13:56:24.832827+0000","last_peered":"2024-09-13T13:56:24.832827+0000","last_clean":"2024-09-13T13:56:24.832827+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:24.832827+0000","last_undegraded":"2024-09-13T13:56:24.832827+0000","last_fullsized":"2024-09-13T13:56:24.832827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:55:29.093390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:55:29.093390+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:56:42.424046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"252'2","reported_seq":509,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:26.709000+0000","last_change":"2024-09-13T13:56:13.367131+0000","last_active":"2024-09-13T13:56:26.709000+0000","last_peered":"2024-09-13T13:56:26.709000+0000","last_clean":"2024-09-13T13:56:26.709000+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:26.709000+0000","last_undegraded":"2024-09-13T13:56:26.709000+0000","last_fullsized":"2024-09-13T13:56:26.709000+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'2","last_scrub_stamp":"2024-09-13T13:56:13.367056+0000","last_deep_scrub":"252'2","last_deep_scrub_stamp":"2024-09-13T13:56:13.367056+0000","last_clean_scrub_stamp":"2024-09-13T13:56:13.367056+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:27.062837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"252'223","reported_seq":882,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:26.709015+0000","last_change":"2024-09-13T13:56:15.464997+0000","last_active":"2024-09-13T13:56:26.709015+0000","last_peered":"2024-09-13T13:56:26.709015+0000","last_clean":"2024-09-13T13:56:26.709015+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:26.709015+0000","last_undegraded":"2024-09-13T13:56:26.709015+0000","last_fullsized":"2024-09-13T13:56:26.709015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'223","last_scrub_stamp":"2024-09-13T13:56:15.464918+0000","last_deep_scrub":"252'223","last_deep_scrub_stamp":"2024-09-13T13:56:15.464918+0000","last_clean_scrub_stamp":"2024-09-13T13:56:15.464918+0000","objects_scrubbed":43,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:42.715967+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":513,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.920658+0000","last_change":"2024-09-13T13:56:08.940307+0000","last_active":"2024-09-13T13:56:25.920658+0000","last_peered":"2024-09-13T13:56:25.920658+0000","last_clean":"2024-09-13T13:56:25.920658+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:25.920658+0000","last_undegraded":"2024-09-13T13:56:25.920658+0000","last_fullsized":"2024-09-13T13:56:25.920658+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_clean_scrub_stamp":"2024-09-13T13:56:08.940224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:27.740256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"251'216","reported_seq":1007,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:24.837418+0000","last_change":"2024-09-13T13:56:12.213629+0000","last_active":"2024-09-13T13:56:24.837418+0000","last_peered":"2024-09-13T13:56:24.837418+0000","last_clean":"2024-09-13T13:56:24.837418+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:24.837418+0000","last_undegraded":"2024-09-13T13:56:24.837418+0000","last_fullsized":"2024-09-13T13:56:24.837418+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'216","last_scrub_stamp":"2024-09-13T13:56:12.213472+0000","last_deep_scrub":"251'216","last_deep_scrub_stamp":"2024-09-13T13:56:12.213472+0000","last_clean_scrub_stamp":"2024-09-13T13:56:12.213472+0000","objects_scrubbed":41,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:39.640531+0000","scrub_duration":550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":515,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:26.708992+0000","last_change":"2024-09-13T13:56:14.367461+0000","last_active":"2024-09-13T13:56:26.708992+0000","last_peered":"2024-09-13T13:56:26.708992+0000","last_clean":"2024-09-13T13:56:26.708992+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:26.708992+0000","last_undegraded":"2024-09-13T13:56:26.708992+0000","last_fullsized":"2024-09-13T13:56:26.708992+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:14.367377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:14.367377+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.367377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:42.414000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"252'191","reported_seq":604,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.328125+0000","last_change":"2024-09-13T13:56:14.354151+0000","last_active":"2024-09-13T13:56:25.328125+0000","last_peered":"2024-09-13T13:56:25.328125+0000","last_clean":"2024-09-13T13:56:25.328125+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:25.328125+0000","last_undegraded":"2024-09-13T13:56:25.328125+0000","last_fullsized":"2024-09-13T13:56:25.328125+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'191","last_scrub_stamp":"2024-09-13T13:56:14.354039+0000","last_deep_scrub":"252'191","last_deep_scrub_stamp":"2024-09-13T13:56:14.354039+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.354039+0000","objects_scrubbed":36,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:43.332285+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"249'3","reported_seq":527,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.918905+0000","last_change":"2024-09-13T13:56:11.084528+0000","last_active":"2024-09-13T13:56:25.918905+0000","last_peered":"2024-09-13T13:56:25.918905+0000","last_clean":"2024-09-13T13:56:25.918905+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:25.918905+0000","last_undegraded":"2024-09-13T13:56:25.918905+0000","last_fullsized":"2024-09-13T13:56:25.918905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'3","last_scrub_stamp":"2024-09-13T13:56:11.084449+0000","last_deep_scrub":"249'3","last_deep_scrub_stamp":"2024-09-13T13:56:11.084449+0000","last_clean_scrub_stamp":"2024-09-13T13:56:11.084449+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:15.235005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"252'187","reported_seq":717,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.766118+0000","last_change":"2024-09-13T13:56:13.440303+0000","last_active":"2024-09-13T13:56:25.766118+0000","last_peered":"2024-09-13T13:56:25.766118+0000","last_clean":"2024-09-13T13:56:25.766118+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:25.766118+0000","last_undegraded":"2024-09-13T13:56:25.766118+0000","last_fullsized":"2024-09-13T13:56:25.766118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'187","last_scrub_stamp":"2024-09-13T13:56:13.440179+0000","last_deep_scrub":"252'187","last_deep_scrub_stamp":"2024-09-13T13:56:13.440179+0000","last_clean_scrub_stamp":"2024-09-13T13:56:13.440179+0000","objects_scrubbed":33,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:41.006539+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":511,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.766929+0000","last_change":"2024-09-13T13:56:10.152629+0000","last_active":"2024-09-13T13:56:25.766929+0000","last_peered":"2024-09-13T13:56:25.766929+0000","last_clean":"2024-09-13T13:56:25.766929+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:25.766929+0000","last_undegraded":"2024-09-13T13:56:25.766929+0000","last_fullsized":"2024-09-13T13:56:25.766929+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_clean_scrub_stamp":"2024-09-13T13:56:10.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:32.489358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":519,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.766976+0000","last_change":"2024-09-13T13:56:14.215132+0000","last_active":"2024-09-13T13:56:25.766976+0000","last_peered":"2024-09-13T13:56:25.766976+0000","last_clean":"2024-09-13T13:56:25.766976+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:25.766976+0000","last_undegraded":"2024-09-13T13:56:25.766976+0000","last_fullsized":"2024-09-13T13:56:25.766976+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:14.215050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:14.215050+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.215050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:43.895454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":506,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.918912+0000","last_change":"2024-09-13T13:56:12.068851+0000","last_active":"2024-09-13T13:56:25.918912+0000","last_peered":"2024-09-13T13:56:25.918912+0000","last_clean":"2024-09-13T13:56:25.918912+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:25.918912+0000","last_undegraded":"2024-09-13T13:56:25.918912+0000","last_fullsized":"2024-09-13T13:56:25.918912+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:12.068763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:12.068763+0000","last_clean_scrub_stamp":"2024-09-13T13:56:12.068763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:13.160999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":513,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T13:56:25.920672+0000","last_change":"2024-09-13T13:56:09.968711+0000","last_active":"2024-09-13T13:56:25.920672+0000","last_peered":"2024-09-13T13:56:25.920672+0000","last_clean":"2024-09-13T13:56:25.920672+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:25.920672+0000","last_undegraded":"2024-09-13T13:56:25.920672+0000","last_fullsized":"2024-09-13T13:56:25.920672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_clean_scrub_stamp":"2024-09-13T13:56:09.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:14.050552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":817,"ondisk_log_size":817,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":1007636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74212,"kb_used_data":43028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297628,"statfs":{"total":96636764160,"available":96560771072,"internally_reserved":0,"allocated":44060672,"data_stored":43764317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":415871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31816,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340024,"statfs":{"total":96636764160,"available":96604184576,"internally_reserved":0,"allocated":647168,"data_stored":416163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280596,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105144,"kb_used_data":73960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266696,"statfs":{"total":96636764160,"available":96529096704,"internally_reserved":0,"allocated":75735040,"data_stored":75373808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":418153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280598,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36576,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335264,"statfs":{"total":96636764160,"available":96599310336,"internally_reserved":0,"allocated":1261568,"data_stored":1007799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280596,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":416148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60280,"kb_used_data":29096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311560,"statfs":{"total":96636764160,"available":96575037440,"internally_reserved":0,"allocated":29794304,"data_stored":29475260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280597,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":417355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66672,"kb_used_data":35488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305168,"statfs":{"total":96636764160,"available":96568492032,"internally_reserved":0,"allocated":36339712,"data_stored":36032343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":675840,"data_stored":418015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247894,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":417003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099473,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63744,"kb_used_data":35504,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308096,"statfs":{"total":96636764160,"available":96571490304,"internally_reserved":0,"allocated":36356096,"data_stored":36033635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247895,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62764,"kb_used_data":31580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309076,"statfs":{"total":96636764160,"available":96572493824,"internally_reserved":0,"allocated":32337920,"data_stored":32026838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247894,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":29088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":29786112,"data_stored":29474305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:35.229 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T13:55:29.093390+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=55, tm_sec=29, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=56, tm_sec=5, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:56:35.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T13:56:41.417 INFO:tasks.ceph.osd.4.smithi027.stderr:2024-09-13T13:56:41.416+0000 7f563b88e640 -1 osd.4 255 handle_osd_map injecting map crc failure 2024-09-13T13:56:43.262 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-13T13:56:43.261+0000 7ffaeece8640 -1 osd.0 256 handle_osd_map injecting map crc failure 2024-09-13T13:56:44.277 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-09-13T13:56:44.277+0000 7fe857f4f640 -1 osd.7 256 handle_osd_map injecting map crc failure 2024-09-13T13:56:55.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:56:55.551 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-13T13:56:55.552 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-13T13:56:55.563 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-13T13:56:54.458757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":140221085,"num_objects":157,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":699,"num_read_kb":27108,"num_write":866,"num_write_kb":266720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":157716943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":934,"ondisk_log_size":934,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":785268,"kb_used_data":285108,"kb_used_omap":36,"kb_used_meta":499867,"kb_avail":1509164172,"statfs":{"total":1546188226560,"available":1545384112128,"internally_reserved":0,"allocated":291950592,"data_stored":287349773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37158,"internal_metadata":511864538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.585202"},"pg_stats":[{"pgid":"1.0","version":"27'112","reported_seq":610,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:46.131074+0000","last_change":"2024-09-13T13:56:46.131074+0000","last_active":"2024-09-13T13:56:46.131074+0000","last_peered":"2024-09-13T13:56:46.131074+0000","last_clean":"2024-09-13T13:56:46.131074+0000","last_became_active":"2024-09-13T13:50:31.733319+0000","last_became_peered":"2024-09-13T13:50:31.733319+0000","last_unstale":"2024-09-13T13:56:46.131074+0000","last_undegraded":"2024-09-13T13:56:46.131074+0000","last_fullsized":"2024-09-13T13:56:46.131074+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'112","last_scrub_stamp":"2024-09-13T13:56:46.130975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:49:47.205190+0000","last_clean_scrub_stamp":"2024-09-13T13:56:46.130975+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:58:07.160157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"252'2","reported_seq":513,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:42.875764+0000","last_change":"2024-09-13T13:56:13.367131+0000","last_active":"2024-09-13T13:56:42.875764+0000","last_peered":"2024-09-13T13:56:42.875764+0000","last_clean":"2024-09-13T13:56:42.875764+0000","last_became_active":"2024-09-13T13:49:50.408567+0000","last_became_peered":"2024-09-13T13:49:50.408567+0000","last_unstale":"2024-09-13T13:56:42.875764+0000","last_undegraded":"2024-09-13T13:56:42.875764+0000","last_fullsized":"2024-09-13T13:56:42.875764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'2","last_scrub_stamp":"2024-09-13T13:56:13.367056+0000","last_deep_scrub":"252'2","last_deep_scrub_stamp":"2024-09-13T13:56:13.367056+0000","last_clean_scrub_stamp":"2024-09-13T13:56:13.367056+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:27.062837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035387400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"252'223","reported_seq":886,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:42.875762+0000","last_change":"2024-09-13T13:56:15.464997+0000","last_active":"2024-09-13T13:56:42.875762+0000","last_peered":"2024-09-13T13:56:42.875762+0000","last_clean":"2024-09-13T13:56:42.875762+0000","last_became_active":"2024-09-13T13:50:50.742952+0000","last_became_peered":"2024-09-13T13:50:50.742952+0000","last_unstale":"2024-09-13T13:56:42.875762+0000","last_undegraded":"2024-09-13T13:56:42.875762+0000","last_fullsized":"2024-09-13T13:56:42.875762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'223","last_scrub_stamp":"2024-09-13T13:56:15.464918+0000","last_deep_scrub":"252'223","last_deep_scrub_stamp":"2024-09-13T13:56:15.464918+0000","last_clean_scrub_stamp":"2024-09-13T13:56:15.464918+0000","objects_scrubbed":43,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:42.715967+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43348398,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":155,"num_read_kb":7582,"num_write":191,"num_write_kb":79835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":47342136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":517,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:43.728437+0000","last_change":"2024-09-13T13:56:08.940307+0000","last_active":"2024-09-13T13:56:43.728437+0000","last_peered":"2024-09-13T13:56:43.728437+0000","last_clean":"2024-09-13T13:56:43.728437+0000","last_became_active":"2024-09-13T13:49:50.370396+0000","last_became_peered":"2024-09-13T13:49:50.370396+0000","last_unstale":"2024-09-13T13:56:43.728437+0000","last_undegraded":"2024-09-13T13:56:43.728437+0000","last_fullsized":"2024-09-13T13:56:43.728437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:08.940224+0000","last_clean_scrub_stamp":"2024-09-13T13:56:08.940224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:27.740256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000449264,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"251'216","reported_seq":1011,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:44.281533+0000","last_change":"2024-09-13T13:56:12.213629+0000","last_active":"2024-09-13T13:56:44.281533+0000","last_peered":"2024-09-13T13:56:44.281533+0000","last_clean":"2024-09-13T13:56:44.281533+0000","last_became_active":"2024-09-13T13:55:30.018298+0000","last_became_peered":"2024-09-13T13:55:30.018298+0000","last_unstale":"2024-09-13T13:56:44.281533+0000","last_undegraded":"2024-09-13T13:56:44.281533+0000","last_fullsized":"2024-09-13T13:56:44.281533+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'216","last_scrub_stamp":"2024-09-13T13:56:12.213472+0000","last_deep_scrub":"251'216","last_deep_scrub_stamp":"2024-09-13T13:56:12.213472+0000","last_clean_scrub_stamp":"2024-09-13T13:56:12.213472+0000","objects_scrubbed":41,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:39.640531+0000","scrub_duration":550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35615618,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":167,"num_read_kb":8661,"num_write":191,"num_write_kb":68306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":65796781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4],"acting":[13,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:42.875762+0000","last_change":"2024-09-13T13:56:14.367461+0000","last_active":"2024-09-13T13:56:42.875762+0000","last_peered":"2024-09-13T13:56:42.875762+0000","last_clean":"2024-09-13T13:56:42.875762+0000","last_became_active":"2024-09-13T13:49:50.229417+0000","last_became_peered":"2024-09-13T13:49:50.229417+0000","last_unstale":"2024-09-13T13:56:42.875762+0000","last_undegraded":"2024-09-13T13:56:42.875762+0000","last_fullsized":"2024-09-13T13:56:42.875762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:14.367377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:14.367377+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.367377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:42.414000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360133,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"252'191","reported_seq":609,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:43.931412+0000","last_change":"2024-09-13T13:56:14.354151+0000","last_active":"2024-09-13T13:56:43.931412+0000","last_peered":"2024-09-13T13:56:43.931412+0000","last_clean":"2024-09-13T13:56:43.931412+0000","last_became_active":"2024-09-13T13:50:50.741514+0000","last_became_peered":"2024-09-13T13:50:50.741514+0000","last_unstale":"2024-09-13T13:56:43.931412+0000","last_undegraded":"2024-09-13T13:56:43.931412+0000","last_fullsized":"2024-09-13T13:56:43.931412+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'191","last_scrub_stamp":"2024-09-13T13:56:14.354039+0000","last_deep_scrub":"252'191","last_deep_scrub_stamp":"2024-09-13T13:56:14.354039+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.354039+0000","objects_scrubbed":36,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:43.332285+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31609575,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":143,"num_read_kb":4156,"num_write":167,"num_write_kb":59643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43987620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"249'3","reported_seq":531,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:44.429716+0000","last_change":"2024-09-13T13:56:11.084528+0000","last_active":"2024-09-13T13:56:44.429716+0000","last_peered":"2024-09-13T13:56:44.429716+0000","last_clean":"2024-09-13T13:56:44.429716+0000","last_became_active":"2024-09-13T13:50:54.734361+0000","last_became_peered":"2024-09-13T13:50:54.734361+0000","last_unstale":"2024-09-13T13:56:44.429716+0000","last_undegraded":"2024-09-13T13:56:44.429716+0000","last_fullsized":"2024-09-13T13:56:44.429716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'3","last_scrub_stamp":"2024-09-13T13:56:11.084449+0000","last_deep_scrub":"249'3","last_deep_scrub_stamp":"2024-09-13T13:56:11.084449+0000","last_clean_scrub_stamp":"2024-09-13T13:56:11.084449+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:15.235005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3","version":"252'187","reported_seq":721,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:45.284320+0000","last_change":"2024-09-13T13:56:13.440303+0000","last_active":"2024-09-13T13:56:45.284320+0000","last_peered":"2024-09-13T13:56:45.284320+0000","last_clean":"2024-09-13T13:56:45.284320+0000","last_became_active":"2024-09-13T13:50:05.074339+0000","last_became_peered":"2024-09-13T13:50:05.074339+0000","last_unstale":"2024-09-13T13:56:45.284320+0000","last_undegraded":"2024-09-13T13:56:45.284320+0000","last_fullsized":"2024-09-13T13:56:45.284320+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'187","last_scrub_stamp":"2024-09-13T13:56:13.440179+0000","last_deep_scrub":"252'187","last_deep_scrub_stamp":"2024-09-13T13:56:13.440179+0000","last_clean_scrub_stamp":"2024-09-13T13:56:13.440179+0000","objects_scrubbed":33,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:41.006539+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29057107,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":148,"num_read_kb":6636,"num_write":170,"num_write_kb":56262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8],"acting":[10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":515,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:42.877800+0000","last_change":"2024-09-13T13:56:10.152629+0000","last_active":"2024-09-13T13:56:42.877800+0000","last_peered":"2024-09-13T13:56:42.877800+0000","last_clean":"2024-09-13T13:56:42.877800+0000","last_became_active":"2024-09-13T13:49:50.227119+0000","last_became_peered":"2024-09-13T13:49:50.227119+0000","last_unstale":"2024-09-13T13:56:42.877800+0000","last_undegraded":"2024-09-13T13:56:42.877800+0000","last_fullsized":"2024-09-13T13:56:42.877800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:10.152540+0000","last_clean_scrub_stamp":"2024-09-13T13:56:10.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:32.489358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039708700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:44.282716+0000","last_change":"2024-09-13T13:56:14.215132+0000","last_active":"2024-09-13T13:56:44.282716+0000","last_peered":"2024-09-13T13:56:44.282716+0000","last_clean":"2024-09-13T13:56:44.282716+0000","last_became_active":"2024-09-13T13:49:50.366211+0000","last_became_peered":"2024-09-13T13:49:50.366211+0000","last_unstale":"2024-09-13T13:56:44.282716+0000","last_undegraded":"2024-09-13T13:56:44.282716+0000","last_fullsized":"2024-09-13T13:56:44.282716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:14.215050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:14.215050+0000","last_clean_scrub_stamp":"2024-09-13T13:56:14.215050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:43.895454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046117400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":510,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:44.429716+0000","last_change":"2024-09-13T13:56:12.068851+0000","last_active":"2024-09-13T13:56:44.429716+0000","last_peered":"2024-09-13T13:56:44.429716+0000","last_clean":"2024-09-13T13:56:44.429716+0000","last_became_active":"2024-09-13T13:49:50.369707+0000","last_became_peered":"2024-09-13T13:49:50.369707+0000","last_unstale":"2024-09-13T13:56:44.429716+0000","last_undegraded":"2024-09-13T13:56:44.429716+0000","last_fullsized":"2024-09-13T13:56:44.429716+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:12.068763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:12.068763+0000","last_clean_scrub_stamp":"2024-09-13T13:56:12.068763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:13.160999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000377024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":517,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T13:56:43.728451+0000","last_change":"2024-09-13T13:56:09.968711+0000","last_active":"2024-09-13T13:56:43.728451+0000","last_peered":"2024-09-13T13:56:43.728451+0000","last_clean":"2024-09-13T13:56:43.728451+0000","last_became_active":"2024-09-13T13:49:50.371841+0000","last_became_peered":"2024-09-13T13:49:50.371841+0000","last_unstale":"2024-09-13T13:56:43.728451+0000","last_undegraded":"2024-09-13T13:56:43.728451+0000","last_fullsized":"2024-09-13T13:56:43.728451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:56:09.968649+0000","last_clean_scrub_stamp":"2024-09-13T13:56:09.968649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T13:57:14.050552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043339000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":139630698,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":613,"num_read_kb":27035,"num_write":719,"num_write_kb":264046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":157126537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300011520,"data_stored":299447566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":817,"ondisk_log_size":817,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1282048,"data_stored":1022368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,5,6,7,8,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74236,"kb_used_data":43052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297604,"statfs":{"total":96636764160,"available":96560746496,"internally_reserved":0,"allocated":44085248,"data_stored":43779049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":688128,"data_stored":430603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280601,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":430895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280600,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105168,"kb_used_data":73984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94266672,"statfs":{"total":96636764160,"available":96529072128,"internally_reserved":0,"allocated":75759616,"data_stored":75388540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":432885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,4,6,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":1286144,"data_stored":1022531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,5,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280600,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":675840,"data_stored":430880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60304,"kb_used_data":29120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311536,"statfs":{"total":96636764160,"available":96575012864,"internally_reserved":0,"allocated":29818880,"data_stored":29489992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,3,5,6,7,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280601,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31856,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":688128,"data_stored":432087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66696,"kb_used_data":35512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94305144,"statfs":{"total":96636764160,"available":96568467456,"internally_reserved":0,"allocated":36364288,"data_stored":36047075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":432747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":431735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":211,"seq":906238099477,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63764,"kb_used_data":35524,"kb_used_omap":12,"kb_used_meta":28211,"kb_avail":94308076,"statfs":{"total":96636764160,"available":96571469824,"internally_reserved":0,"allocated":36376576,"data_stored":36048073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13234,"internal_metadata":28888142},"hb_peers":[2,3,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247899,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62784,"kb_used_data":31600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309056,"statfs":{"total":96636764160,"available":96572473344,"internally_reserved":0,"allocated":32358400,"data_stored":32041276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":23,"seq":98784247898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64456,"kb_used_data":29112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307384,"statfs":{"total":96636764160,"available":96570761216,"internally_reserved":0,"allocated":29810688,"data_stored":29489037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,6,7,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20221952,"data_stored":20186170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75091968,"data_stored":74957973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31674368,"data_stored":31609575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29118464,"data_stored":29057107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":35684352,"data_stored":35615618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":43417600,"data_stored":43348398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:56:55.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T13:56:55.885 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T13:56:55.885 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T13:56:55.885 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T13:56:55.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:55.920 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T13:56:55.921 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T13:56:55.921 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:55.991 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T13:56:55.992 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-13T13:56:55.992 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.061 INFO:tasks.ceph.osd.8:Stopped 2024-09-13T13:56:56.062 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-13T13:56:56.062 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.132 INFO:tasks.ceph.osd.12:Stopped 2024-09-13T13:56:56.132 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T13:56:56.132 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.164 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T13:56:56.165 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T13:56:56.165 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.232 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T13:56:56.233 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-13T13:56:56.233 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.299 INFO:tasks.ceph.osd.9:Stopped 2024-09-13T13:56:56.299 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-13T13:56:56.299 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.366 INFO:tasks.ceph.osd.13:Stopped 2024-09-13T13:56:56.367 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T13:56:56.367 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.406 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T13:56:56.406 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T13:56:56.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.485 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T13:56:56.486 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-13T13:56:56.486 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.559 INFO:tasks.ceph.osd.10:Stopped 2024-09-13T13:56:56.559 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-13T13:56:56.559 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.636 INFO:tasks.ceph.osd.14:Stopped 2024-09-13T13:56:56.636 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T13:56:56.637 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.671 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T13:56:56.671 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T13:56:56.671 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.743 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T13:56:56.743 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-13T13:56:56.743 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.818 INFO:tasks.ceph.osd.11:Stopped 2024-09-13T13:56:56.819 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-13T13:56:56.819 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.893 INFO:tasks.ceph.osd.15:Stopped 2024-09-13T13:56:56.893 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T13:56:56.893 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T13:56:56.893 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.923 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T13:56:56.923 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T13:56:56.923 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.966 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T13:56:56.966 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T13:56:56.966 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T13:56:56.967 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.980 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T13:56:56.980 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T13:56:56.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:56.994 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T13:56:56.995 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T13:56:56.995 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:56:57.009 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T13:56:57.009 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T13:56:57.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T13:56:57.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:56:57.057 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T13:56:57.273 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:56:57.273 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T13:56:57.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:56:57.356 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-13T13:56:57.429 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:56:57.430 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-13T13:56:57.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:56:57.480 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T13:56:57.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:56:57.606 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T13:56:57.700 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:56:57.701 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-13T13:56:57.760 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:56:57.761 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-13T13:56:57.815 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:56:57.815 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T13:56:58.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:56:58.221 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T13:56:58.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:56:58.283 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-13T13:56:58.360 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:56:58.360 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-13T13:56:58.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:56:58.425 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T13:56:58.551 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:56:58.551 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T13:56:58.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:56:58.630 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-13T13:56:58.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:56:58.688 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-13T13:56:58.745 INFO:tasks.ceph:Archiving mon data... 2024-09-13T13:56:58.746 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/data/mon.a.tgz 2024-09-13T13:56:58.747 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-13T13:56:58.763 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.6dkdI6JUUC 2024-09-13T13:56:58.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6dkdI6JUUC 2024-09-13T13:56:58.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.6dkdI6JUUC 2024-09-13T13:56:59.019 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.6dkdI6JUUC is 475KB 2024-09-13T13:56:59.117 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.6dkdI6JUUC 2024-09-13T13:56:59.132 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/data/mon.b.tgz 2024-09-13T13:56:59.132 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-13T13:56:59.151 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.EDJVAIyzIn 2024-09-13T13:56:59.151 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.EDJVAIyzIn 2024-09-13T13:56:59.438 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.EDJVAIyzIn 2024-09-13T13:56:59.522 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.EDJVAIyzIn is 484KB 2024-09-13T13:56:59.619 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.EDJVAIyzIn 2024-09-13T13:56:59.635 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/data/mon.c.tgz 2024-09-13T13:56:59.635 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-13T13:56:59.655 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.Ke1XlYeVeJ 2024-09-13T13:56:59.655 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Ke1XlYeVeJ 2024-09-13T13:56:59.830 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.Ke1XlYeVeJ 2024-09-13T13:56:59.922 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.Ke1XlYeVeJ is 484KB 2024-09-13T13:57:00.019 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.Ke1XlYeVeJ 2024-09-13T13:57:00.038 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T13:57:00.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:57:00.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:57:00.043 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:57:00.079 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:57:00.127 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:57:00.143 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:57:00.166 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:57:00.185 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:57:00.185 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T13:57:00.187 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi027/crash 2024-09-13T13:57:00.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:57:00.217 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi100/crash 2024-09-13T13:57:00.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:57:00.244 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi122/crash 2024-09-13T13:57:00.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:57:00.274 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi159/crash 2024-09-13T13:57:00.275 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:57:00.407 INFO:tasks.ceph:Compressing logs... 2024-09-13T13:57:00.430 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:57:00.433 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:57:00.436 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:57:00.438 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:57:00.462 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T13:57:00.462 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-13T13:57:00.462 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-13T13:57:00.462 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T13:57:00.462 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T13:57:00.463 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T13:57:00.463 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T13:57:00.464 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T13:57:00.471 INFO:teuthology.orchestra.run.smithi100.stderr: 89.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-13T13:57:00.478 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-13T13:57:00.479 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T13:57:00.479 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.6.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T13:57:00.479 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.log: 89.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T13:57:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38319.log 2024-09-13T13:57:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33417.log 2024-09-13T13:57:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.0.38319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33257.log 2024-09-13T13:57:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33417.log: 89.8% -- replaced with /var/log/ceph/ceph-client.0.38319.log.gz 2024-09-13T13:57:00.491 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38296.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33417.log.gz 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33097.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33257.log.gz 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.38296.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38296.log.gz 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33097.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33097.log.gz -- 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr: /var/log/ceph/ceph-client.admin.32926.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33137.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33497.log 2024-09-13T13:57:00.492 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33137.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33137.log.gz 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33457.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33497.log.gz 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33057.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33337.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33457.log.gz 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33057.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.33057.log.gz -- 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr: /var/log/ceph/ceph-osd.11.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32968.log 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33337.log.gz 2024-09-13T13:57:00.493 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33177.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32968.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32968.log.gz 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33177.log.gz 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32886.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.32886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33017.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32886.log.gz 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33377.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33297.log 2024-09-13T13:57:00.494 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33017.log.gz 2024-09-13T13:57:00.495 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33377.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T13:57:00.495 INFO:teuthology.orchestra.run.smithi159.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33377.log.gz 2024-09-13T13:57:00.495 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33297.log.gz 2024-09-13T13:57:00.585 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.3.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52777.log 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39316.log 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.52777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46699.log 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52777.log.gz 2024-09-13T13:57:00.648 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39316.log.gz 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46699.log: gzip -5 --verbose 19.5% -- -- replaced with /var/log/ceph/ceph-client.admin.46699.log.gz /var/log/ceph/ceph-client.admin.43039.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39351.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62046.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59859.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39351.log.gz 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45376.log 2024-09-13T13:57:00.649 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62046.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59859.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.62677.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59859.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58140.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45376.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55002.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62677.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43121.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55002.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.39240.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55002.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58896.log 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.43121.log.gz 2024-09-13T13:57:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60912.log 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45778.log 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58896.log.gz 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58051.log 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60912.log.gz 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61542.log 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.45778.log.gz 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-09-13T13:57:00.651 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58051.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58308.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61542.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58308.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59400.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61500.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59400.log.gz 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45913.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62130.log 2024-09-13T13:57:00.652 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61500.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60346.log 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45913.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62130.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60346.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58686.log 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62340.log 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58686.log.gz 2024-09-13T13:57:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59358.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59064.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62340.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52717.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59358.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59064.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50025.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59064.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57020.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr: 19.0% -- replaced with /var/log/ceph/ceph-client.admin.52717.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/objectstore_tool.50025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58093.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57020.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58093.log.gz 2024-09-13T13:57:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59819.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46039.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59819.log.gz 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50891.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.46039.log.gz 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50891.log.gz 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62550.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56193.log 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-09-13T13:57:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62550.log.gz 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56193.log.gz 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44387.log: gzip -5 --verbose -- 19.1% /var/log/ceph/ceph-client.admin.60600.log -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61962.log 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60600.log.gz 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39708.log 2024-09-13T13:57:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61962.log.gz 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39708.log.gz 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55248.log 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47147.log 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55248.log.gz 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60459.log 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54505.log 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47147.log.gz 2024-09-13T13:57:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60459.log.gz 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60489.log 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.54505.log.gz 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60489.log.gz 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38024.log 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33243.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz -5 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55115.log 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-09-13T13:57:00.658 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38024.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59836.log 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55115.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49865.log 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59836.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59856.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60508.log 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49865.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-09-13T13:57:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60493.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.50025.log.gz 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58854.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60508.log.gz 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48809.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60493.log.gz 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47598.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58854.log.gz 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48340.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48809.log.gz 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44041.log 2024-09-13T13:57:00.660 INFO:teuthology.orchestra.run.smithi027.stderr: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.47598.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49322.log 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48340.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60525.log 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr: 18.4% -- replaced with /var/log/ceph/ceph-client.admin.44041.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38640.log 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.49322.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58350.log 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60525.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38640.log.gz 2024-09-13T13:57:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58350.log.gz 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59833.log 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gz 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58560.log 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-09-13T13:57:00.662 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59833.log.gz 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58560.log.gz 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56978.log 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56978.log.gz 2024-09-13T13:57:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55927.log 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54751.log 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55927.log.gz 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62172.log 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr: 19.5% -- replaced with /var/log/ceph/ceph-client.admin.54751.log.gz 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62172.log.gz 2024-09-13T13:57:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46395.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58182.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46395.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54638.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58182.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55409.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42884.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 20.5% -- replaced with /var/log/ceph/ceph-client.admin.54638.log.gz 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56274.log 2024-09-13T13:57:00.665 INFO:teuthology.orchestra.run.smithi027.stderr: 19.0% -- replaced with /var/log/ceph/ceph-client.admin.55409.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.42884.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54709.log 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42884.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.56274.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54709.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42219.log 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr: 51.6% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42219.log: 19.5% -- replaced with /var/log/ceph/ceph-client.admin.42219.log.gz 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44262.log 2024-09-13T13:57:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40409.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59274.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44262.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40409.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62256.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59274.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52901.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62256.log.gz 2024-09-13T13:57:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58602.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52901.log.gz 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62088.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56487.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58602.log.gz 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62088.log.gz 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61710.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 19.5% -- replaced with /var/log/ceph/ceph-client.admin.56487.log.gz 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-09-13T13:57:00.668 INFO:teuthology.orchestra.run.smithi027.stderr: 20.8% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61794.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61710.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38620.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61374.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61794.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44857.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61080.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61374.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39290.log 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.44857.log.gz 2024-09-13T13:57:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61080.log.gz 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr: 28.0% -- replaced with /var/log/ceph/ceph-client.admin.39290.log.gz 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz /var/log/ceph/ceph-client.admin.54596.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43322.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54596.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54596.log.gz 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43322.log.gz 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60954.log 2024-09-13T13:57:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.60954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60954.log.gz 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39083.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38080.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53826.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38080.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38080.log.gz -5 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45914.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58224.log 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53826.log.gz 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45914.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45914.log.gz 2024-09-13T13:57:00.671 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56080.log 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58224.log.gz 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58938.log 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.56080.log.gz 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49750.log 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53926.log 0.0% 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58938.log.gz 2024-09-13T13:57:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54119.log 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49750.log.gz 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53926.log.gz 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59022.log 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54119.log: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.54119.log.gz 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2024-09-13T13:57:00.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56732.log 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43551.log 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59022.log.gz 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56732.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56732.log.gz -5 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45128.log 2024-09-13T13:57:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43551.log: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.43551.log.gz 2024-09-13T13:57:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47652.log 2024-09-13T13:57:00.675 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2024-09-13T13:57:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55674.log 2024-09-13T13:57:00.675 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45128.log.gz 2024-09-13T13:57:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38322.log 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47652.log.gz 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55674.log: 20.4% -- replaced with /var/log/ceph/ceph-client.admin.55674.log.gz 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55967.log 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38322.log.gz 2024-09-13T13:57:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48141.log 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61038.log 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.55967.log.gz 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43571.log 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48141.log.gz 2024-09-13T13:57:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61038.log.gz 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44346.log 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43571.log.gz 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61248.log 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-09-13T13:57:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44931.log 2024-09-13T13:57:00.679 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44346.log.gz 2024-09-13T13:57:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49278.log 2024-09-13T13:57:00.679 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61248.log.gz 2024-09-13T13:57:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58392.log 2024-09-13T13:57:00.679 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44931.log.gz 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47330.log 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49278.log.gz 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53348.log 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58392.log.gz 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58980.log 2024-09-13T13:57:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47330.log.gz 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58009.log 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.53348.log.gz 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45978.log 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58980.log.gz 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61458.log 2024-09-13T13:57:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58009.log.gz 2024-09-13T13:57:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45978.log.gz 2024-09-13T13:57:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44108.log 2024-09-13T13:57:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38621.log 2024-09-13T13:57:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61458.log.gz 2024-09-13T13:57:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61836.log 2024-09-13T13:57:00.683 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44108.log.gz 2024-09-13T13:57:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38621.log.gz 2024-09-13T13:57:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46479.log 2024-09-13T13:57:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61122.log 2024-09-13T13:57:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61836.log.gz 2024-09-13T13:57:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46479.log.gz 2024-09-13T13:57:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61206.log 2024-09-13T13:57:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61878.log 2024-09-13T13:57:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61122.log.gz 2024-09-13T13:57:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59835.log 2024-09-13T13:57:00.685 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61206.log.gz 2024-09-13T13:57:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44304.log 2024-09-13T13:57:00.685 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61878.log.gz 2024-09-13T13:57:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46616.log 2024-09-13T13:57:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59835.log.gz 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40204.log 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44304.log.gz 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53560.log 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46616.log.gz 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59853.log 2024-09-13T13:57:00.686 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40204.log.gz 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39359.log 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53560.log.gz 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58266.log 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59853.log.gz 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39359.log.gz 2024-09-13T13:57:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55521.log 2024-09-13T13:57:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-09-13T13:57:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58266.log.gz 2024-09-13T13:57:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48852.log 2024-09-13T13:57:00.688 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55521.log.gz 2024-09-13T13:57:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44641.log 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41941.log 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48852.log.gz 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56040.log 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.44641.log.gz 2024-09-13T13:57:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41941.log.gz 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56040.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.56040.log.gz 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.40244.log 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45273.log 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-09-13T13:57:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52362.log 2024-09-13T13:57:00.691 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-09-13T13:57:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2024-09-13T13:57:00.691 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45273.log.gz 2024-09-13T13:57:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48298.log 2024-09-13T13:57:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52362.log: 19.5% -- replaced with /var/log/ceph/ceph-client.admin.52362.log.gz 2024-09-13T13:57:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46478.log 2024-09-13T13:57:00.692 INFO:teuthology.orchestra.run.smithi027.stderr: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2024-09-13T13:57:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-09-13T13:57:00.692 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48298.log.gz 2024-09-13T13:57:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2024-09-13T13:57:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46478.log: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.46478.log.gz 2024-09-13T13:57:00.693 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-09-13T13:57:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48595.log 2024-09-13T13:57:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58476.log 2024-09-13T13:57:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2024-09-13T13:57:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40124.log 2024-09-13T13:57:00.697 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48595.log.gz 2024-09-13T13:57:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58518.log 2024-09-13T13:57:00.697 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58476.log.gz 2024-09-13T13:57:00.697 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-09-13T13:57:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40124.log.gz 2024-09-13T13:57:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61668.log 2024-09-13T13:57:00.698 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58518.log.gz 2024-09-13T13:57:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-09-13T13:57:00.698 INFO:teuthology.orchestra.run.smithi027.stderr: 18.4% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61668.log.gz 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40285.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-09-13T13:57:00.699 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45673.log 2024-09-13T13:57:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61584.log 2024-09-13T13:57:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-09-13T13:57:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2024-09-13T13:57:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55562.log 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61584.log.gz 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61626.log 2024-09-13T13:57:00.701 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60491.log 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61626.log.gz 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60491.log.gz 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-09-13T13:57:00.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45098.log 2024-09-13T13:57:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-09-13T13:57:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48737.log 2024-09-13T13:57:00.703 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-09-13T13:57:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41040.log 2024-09-13T13:57:00.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45098.log: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.45098.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48737.log: gzip -5 --verbose -- 19.0% -- replaced with /var/log/ceph/ceph-client.admin.48737.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.52830.log 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41040.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47148.log 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59816.log 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47148.log: gzip 18.4% -- replaced with /var/log/ceph/ceph-client.admin.47148.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46574.log 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59816.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27140.log 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59816.log.gz 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46574.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46574.log.gz -5 2024-09-13T13:57:00.704 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58728.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47968.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.27140.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27140.log.gz 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58728.log.gz 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60464.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr: 19.9% -- replaced with /var/log/ceph/ceph-client.admin.47968.log.gz 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55634.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr: 19.7% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60464.log.gz 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45057.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38660.log 2024-09-13T13:57:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55634.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46913.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45057.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38660.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61416.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46913.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43707.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61920.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61416.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43707.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.43707.log.gz 2024-09-13T13:57:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61920.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59442.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39314.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53307.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59442.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54303.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39314.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53307.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53307.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.40367.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54303.log.gz 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60469.log 2024-09-13T13:57:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40367.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44816.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60517.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60469.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61164.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44816.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40084.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60517.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49947.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61164.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45015.log 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40084.log.gz 2024-09-13T13:57:00.708 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58770.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49947.log.gz 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.45015.log: --verbose -- /var/log/ceph/ceph-client.admin.59867.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45015.log.gz 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58770.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58770.log.gz 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59867.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.59867.log.gz /var/log/ceph/ceph-client.admin.60447.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49395.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53194.log 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60447.log.gz 2024-09-13T13:57:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.51553.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49395.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62592.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53194.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/objectstore_tool.51553.log: /var/log/ceph/ceph-client.admin.53776.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62592.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60269.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53776.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53776.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59526.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60269.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38664.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59526.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59526.log.gz 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.59866.log 2024-09-13T13:57:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38664.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48382.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59866.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59866.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61752.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48382.log: 20.8% -- replaced with /var/log/ceph/ceph-client.admin.48382.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47786.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61290.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61752.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47786.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47786.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.39213.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61290.log.gz 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59190.log 2024-09-13T13:57:00.711 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.39213.log: --verbose -- /var/log/ceph/ceph-client.admin.45824.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59190.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60996.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.45824.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52322.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45824.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60340.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60996.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.52322.log: -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52322.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60340.log.gz 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40844.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59106.log 2024-09-13T13:57:00.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40844.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54190.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62004.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59106.log.gz 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54190.log.gz 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58812.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45538.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58812.log: /var/log/ceph/ceph-client.admin.62004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58812.log.gz 25.6% 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62004.log.gz 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59852.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45538.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45538.log.gz 2024-09-13T13:57:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60506.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59852.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60506.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38105.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40676.log: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42609.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38105.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59860.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42609.log: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.42609.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58644.log 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59860.log: 0.0% 93.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-13T13:57:00.714 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59860.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56559.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38628.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58644.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56559.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50680.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49907.log: /var/log/ceph/ceph-client.admin.38628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38628.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55156.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49637.log 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0%/var/log/ceph/objectstore_tool.50680.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T13:57:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55156.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.55156.log.gz 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56600.log 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41984.log 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr: 17.8% -- replaced with /var/log/ceph/ceph-client.admin.49637.log.gz 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56600.log: 20.3% -- replaced with /var/log/ceph/ceph-client.admin.56600.log.gz 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48256.log 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38642.log 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr: 19.2% -- replaced with /var/log/ceph/ceph-client.admin.41984.log.gz 2024-09-13T13:57:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48256.log.gz 2024-09-13T13:57:00.717 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32975.log 2024-09-13T13:57:00.717 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38196.log 2024-09-13T13:57:00.717 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38642.log.gz 2024-09-13T13:57:00.717 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32975.log.gz 2024-09-13T13:57:00.717 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46193.log 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38196.log.gz 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46193.log.gz 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59316.log 2024-09-13T13:57:00.718 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62424.log 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59316.log.gz 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60458.log 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62424.log.gz 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44581.log 2024-09-13T13:57:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.60458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60458.log.gz 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59868.log 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44581.log.gz 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59868.log.gz 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51483.log 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-09-13T13:57:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62637.log 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51483.log: 19.0% -- replaced with /var/log/ceph/ceph-client.admin.51483.log.gz 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62382.log 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62637.log.gz 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46993.log 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62382.log.gz 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54232.log 2024-09-13T13:57:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46993.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.46993.log.gz 2024-09-13T13:57:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2024-09-13T13:57:00.722 INFO:teuthology.orchestra.run.smithi027.stderr: 93.6% -- replaced with /var/log/ceph/objectstore_tool.51553.log.gz/var/log/ceph/ceph-client.admin.54232.log: 2024-09-13T13:57:00.722 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54232.log.gz 2024-09-13T13:57:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52447.log 2024-09-13T13:57:00.722 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49207.log: 19.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56446.log 2024-09-13T13:57:00.723 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2024-09-13T13:57:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59826.log 2024-09-13T13:57:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52447.log.gz/var/log/ceph/ceph-client.admin.56446.log: 2024-09-13T13:57:00.723 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56446.log.gz 2024-09-13T13:57:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40934.log 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59826.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61332.log 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40934.log: /var/log/ceph/ceph-client.admin.45737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40934.log.gz 2024-09-13T13:57:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52943.log 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.61332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61332.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40555.log 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38611.log: /var/log/ceph/ceph-client.admin.52943.log: 0.0% 20.3% -- replaced with /var/log/ceph/ceph-client.admin.52943.log.gz 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56814.log 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40555.log.gzgzip 2024-09-13T13:57:00.725 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42057.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58434.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42057.log: /var/log/ceph/ceph-client.admin.56814.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56814.log.gz 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42057.log.gz 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38639.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.58434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58434.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44930.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38639.log.gz 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62508.log 2024-09-13T13:57:00.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44930.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50089.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62466.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62508.log.gz/var/log/ceph/ceph-client.admin.50089.log: 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr: 19.0% -- replaced with /var/log/ceph/ceph-client.admin.50089.log.gz 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62298.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56896.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62466.log.gz 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48183.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.56896.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56896.log.gz 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62298.log.gz 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48183.log: 18.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-13T13:57:00.727 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48183.log.gz 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62214.log 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59858.log: /var/log/ceph/ceph-client.admin.41442.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.62214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62214.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45653.log 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38651.log: /var/log/ceph/ceph-client.admin.43666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59484.log 2024-09-13T13:57:00.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45653.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45579.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45653.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59232.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59484.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47713.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.45579.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59232.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.47713.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53713.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53713.log: /var/log/ceph/ceph-client.admin.39782.log: 18.2% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53713.log.gz 2024-09-13T13:57:00.729 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.50405.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42965.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55043.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/objectstore_tool.50405.log: /var/log/ceph/ceph-client.admin.42965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42965.log.gz 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44195.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55043.log: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.55043.log.gz 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53236.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44195.log.gz 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53236.log: 19.0% -- replaced with /var/log/ceph/ceph-client.admin.53236.log.gz 2024-09-13T13:57:00.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33363.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48941.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33363.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57062.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48941.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59148.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.57062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57062.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49792.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.59148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59148.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49792.log: 18.6% -- replaced with /var/log/ceph/ceph-client.admin.49792.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-09-13T13:57:00.731 INFO:teuthology.orchestra.run.smithi027.stderr: 94.4% -- replaced with /var/log/ceph/objectstore_tool.50405.log.gz 2024-09-13T13:57:00.732 INFO:teuthology.orchestra.run.smithi027.stderr: 94.4% -- replaced with /var/log/ceph/objectstore_tool.50680.log.gz 2024-09-13T13:57:00.732 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T13:57:00.808 INFO:teuthology.orchestra.run.smithi100.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T13:57:00.996 INFO:teuthology.orchestra.run.smithi100.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T13:57:01.075 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T13:57:01.160 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-13T13:57:01.167 INFO:teuthology.orchestra.run.smithi122.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-13T13:57:01.186 INFO:teuthology.orchestra.run.smithi159.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-13T13:57:01.207 INFO:teuthology.orchestra.run.smithi159.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T13:57:01.236 INFO:teuthology.orchestra.run.smithi100.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-13T13:57:01.246 INFO:teuthology.orchestra.run.smithi159.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-13T13:57:01.319 INFO:teuthology.orchestra.run.smithi100.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T13:57:01.392 INFO:teuthology.orchestra.run.smithi027.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T13:57:01.452 INFO:teuthology.orchestra.run.smithi027.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T13:57:01.460 INFO:teuthology.orchestra.run.smithi159.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-13T13:57:01.462 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-09-13T13:57:01.462 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m1.006s 2024-09-13T13:57:01.462 INFO:teuthology.orchestra.run.smithi159.stderr:user 0m3.231s 2024-09-13T13:57:01.462 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m0.128s 2024-09-13T13:57:01.586 INFO:teuthology.orchestra.run.smithi100.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-13T13:57:01.588 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-09-13T13:57:01.588 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m1.140s 2024-09-13T13:57:01.589 INFO:teuthology.orchestra.run.smithi100.stderr:user 0m3.540s 2024-09-13T13:57:01.589 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m0.135s 2024-09-13T13:57:01.635 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T13:57:01.693 INFO:teuthology.orchestra.run.smithi122.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-13T13:57:01.728 INFO:teuthology.orchestra.run.smithi122.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-13T13:57:01.730 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-13T13:57:01.730 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m1.276s 2024-09-13T13:57:01.730 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m4.446s 2024-09-13T13:57:01.730 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.155s 2024-09-13T13:57:02.098 INFO:teuthology.orchestra.run.smithi027.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-13T13:57:02.099 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-13T13:57:02.100 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m1.651s 2024-09-13T13:57:02.100 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m4.744s 2024-09-13T13:57:02.100 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.357s 2024-09-13T13:57:02.100 INFO:tasks.ceph:Archiving logs... 2024-09-13T13:57:02.100 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi027/log 2024-09-13T13:57:02.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:57:03.120 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi100/log 2024-09-13T13:57:03.121 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:57:03.709 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi122/log 2024-09-13T13:57:03.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:57:04.450 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi159/log 2024-09-13T13:57:04.451 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:57:04.991 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T13:57:05.003 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T13:57:05.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:57:05.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:57:05.009 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:57:05.011 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:57:05.160 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T13:57:05.168 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T13:57:05.169 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:57:05.171 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:57:05.174 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:57:05.176 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:57:05.186 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:57:05.188 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:57:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:57:05.191 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:57:05.191 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-13T13:57:05.191 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 -4770ns[+7344ns] +/- 40ms 2024-09-13T13:57:05.191 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -523us[ -523us] +/- 70ms 2024-09-13T13:57:05.192 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +633us[ +646us] +/- 70ms 2024-09-13T13:57:05.192 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:57:05.193 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:57:05.193 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:57:05.193 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-13T13:57:05.193 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 -9121ns[+8826ns] +/- 40ms 2024-09-13T13:57:05.193 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 -524us[ -524us] +/- 70ms 2024-09-13T13:57:05.194 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +616us[ +634us] +/- 71ms 2024-09-13T13:57:05.194 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:57:05.195 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:57:05.195 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-13T13:57:05.195 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 -6504ns[ +19us] +/- 40ms 2024-09-13T13:57:05.196 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 -523us[ -523us] +/- 70ms 2024-09-13T13:57:05.196 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +610us[ +610us] +/- 71ms 2024-09-13T13:57:05.196 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:57:05.197 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:57:05.197 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-09-13T13:57:05.198 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 -28us[ -24us] +/- 40ms 2024-09-13T13:57:05.198 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -532us[ -528us] +/- 70ms 2024-09-13T13:57:05.198 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +620us[ +624us] +/- 70ms 2024-09-13T13:57:05.198 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:57:05.198 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T13:57:05.210 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T13:57:05.211 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T13:57:05.219 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:57:05.316 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:57:05.390 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:57:05.467 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:57:05.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:57:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:05.573 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:57:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:05.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:57:05.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:05.638 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:57:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:05.669 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T13:57:05.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T13:57:05.755 INFO:teuthology.task.internal:Duration was 907.733165 seconds 2024-09-13T13:57:05.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T13:57:05.768 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T13:57:05.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:57:05.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:57:05.773 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:57:05.776 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:57:05.811 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:05.815 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:05.816 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:05.821 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:57:06.306 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T13:57:06.306 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-09-13T13:57:06.307 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:57:06.333 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-09-13T13:57:06.333 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:57:06.360 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-09-13T13:57:06.360 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:57:06.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-09-13T13:57:06.396 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:57:06.422 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T13:57:06.423 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:57:06.425 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:57:06.428 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:57:06.439 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:57:06.490 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T13:57:06.491 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:57:06.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:57:06.501 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:57:06.509 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:57:06.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T13:57:06.655 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T13:57:06.656 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:57:06.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:57:06.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:57:06.731 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:57:06.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T13:57:06.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.771 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.773 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.776 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.793 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-09-13T13:57:06.794 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-09-13T13:57:06.796 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-09-13T13:57:06.820 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-09-13T13:57:06.845 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:06.868 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:06.883 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:06.898 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:57:06.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T13:57:06.924 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T13:57:06.925 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi027 2024-09-13T13:57:06.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:57:06.961 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi100 2024-09-13T13:57:06.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:57:06.996 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi122 2024-09-13T13:57:06.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:57:07.030 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903513/remote/smithi159 2024-09-13T13:57:07.030 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:57:07.063 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T13:57:07.063 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:07.066 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:07.069 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:07.071 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:57:07.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T13:57:07.124 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T13:57:07.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T13:57:07.136 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T13:57:07.136 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:57:07.139 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:57:07.141 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:57:07.143 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:57:07.152 INFO:teuthology.orchestra.run.smithi027.stdout: 262173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:57 /home/ubuntu/cephtest 2024-09-13T13:57:07.153 INFO:teuthology.orchestra.run.smithi100.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:57 /home/ubuntu/cephtest 2024-09-13T13:57:07.156 INFO:teuthology.orchestra.run.smithi122.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:57 /home/ubuntu/cephtest 2024-09-13T13:57:07.169 INFO:teuthology.orchestra.run.smithi159.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:57 /home/ubuntu/cephtest 2024-09-13T13:57:07.170 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T13:57:07.183 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T13:57:07.217 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} duration: 907.7331647872925 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T13:57:07.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:57:07.325 INFO:teuthology.run:pass